[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651852476208-21193-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:17:32 dist-test-slave-c9qh krb5kdc[21200](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:17:32 dist-test-slave-c9qh krb5kdc[21200](info): set up 2 sockets
Sep 29 03:17:32 dist-test-slave-c9qh krb5kdc[21200](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260929 03:17:33.521948 21193 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651852476208-21193-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651852476208-21193-0/krb5kdc/test-user.keytab.
Sep 29 03:17:33 dist-test-slave-c9qh krb5kdc[21200](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651853, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:17:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:17:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.178.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.178.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.178.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.126.keytab.
Entry for principal kudu/127.20.178.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.126.keytab.
I20260929 03:17:33.639828 21193 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuoaHbX/build/release/bin/kudu
/tmp/dist-test-taskuoaHbX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651852476208-21193-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651852476208-21193-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651852476208-21193-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651852476208-21193-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.178.126:33269
--webserver_interface=127.20.178.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.126.keytab
--principal=kudu/127.20.178.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.178.84:42513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.178.126:33269
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651852476208-21193-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651852476208-21193-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651852476208-21193-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:17:33.719080 21213 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:17:33.719309 21213 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:17:33.719347 21213 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:17:33.720479 21213 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:17:33.720543 21213 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:17:33.720566 21213 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:17:33.720588 21213 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:17:33.720616 21213 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:17:33.722100 21213 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.178.84:42513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651852476208-21193-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651852476208-21193-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.178.126:33269
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.126.keytab
--principal=kudu/127.20.178.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.178.126:33269
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651852476208-21193-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.178.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21213
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651852476208-21193-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:17:33.722393 21213 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:17:33.722625 21213 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:17:33.727619 21218 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:17:33.727592 21219 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:17:33.727806 21213 server_base.cc:1034] running on GCE node
W20260929 03:17:33.727746 21221 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:17:33.728130 21213 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:17:33.728420 21213 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:17:33.729583 21213 hybrid_clock.cc:648] HybridClock initialized: now 1790651853729567 us; error 26 us; skew 500 ppm
Sep 29 03:17:33 dist-test-slave-c9qh krb5kdc[21200](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651853, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:17:33.731951 21213 init.cc:377] Logged in from keytab as kudu/127.20.178.126@KRBTEST.COM (short username kudu)
I20260929 03:17:33.732362 21213 webserver.cc:460] Webserver started at http://127.20.178.126:34149/ using document root <none> and password file <none>
I20260929 03:17:33.732684 21213 fs_manager.cc:359] Metadata directory not provided
I20260929 03:17:33.732756 21213 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:17:33.732913 21213 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:17:33.734036 21213 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651852476208-21193-0/minicluster-data/master-0/data/instance:
uuid: "76abf8500dd540fabb1acfe5a3c641f5"
format_stamp: "Formatted at 2026-09-29 03:17:33 on dist-test-slave-c9qh"
server_key: "1d62793ff8e6b149d7af00041ca83d48"
server_key_iv: "d0082b9e701190c1331f4bbe737cc614"
server_key_version: "encryptionkey@0"
I20260929 03:17:33.734411 21213 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651852476208-21193-0/minicluster-data/master-0/wal/instance:
uuid: "76abf8500dd540fabb1acfe5a3c641f5"
format_stamp: "Formatted at 2026-09-29 03:17:33 on dist-test-slave-c9qh"
server_key: "1d62793ff8e6b149d7af00041ca83d48"
server_key_iv: "d0082b9e701190c1331f4bbe737cc614"
server_key_version: "encryptionkey@0"
I20260929 03:17:33.735833 21213 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 03:17:33.736603 21228 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:17:33.736801 21213 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 03:17:33.736900 21213 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651852476208-21193-0/minicluster-data/master-0/data,/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651852476208-21193-0/minicluster-data/master-0/wal
uuid: "76abf8500dd540fabb1acfe5a3c641f5"
format_stamp: "Formatted at 2026-09-29 03:17:33 on dist-test-slave-c9qh"
server_key: "1d62793ff8e6b149d7af00041ca83d48"
server_key_iv: "d0082b9e701190c1331f4bbe737cc614"
server_key_version: "encryptionkey@0"
I20260929 03:17:33.736982 21213 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651852476208-21193-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651852476208-21193-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651852476208-21193-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:17:33.751008 21213 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:17:33.752821 21213 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:17:33.752971 21213 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:17:33.756785 21213 rpc_server.cc:307] RPC server started. Bound to: 127.20.178.126:33269
I20260929 03:17:33.756832 21280 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.178.126:33269 every 8 connection(s)
I20260929 03:17:33.757180 21213 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651852476208-21193-0/minicluster-data/master-0/data/info.pb
I20260929 03:17:33.758072 21281 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:17:33.760229 21281 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76abf8500dd540fabb1acfe5a3c641f5: Bootstrap starting.
I20260929 03:17:33.760824 21281 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 76abf8500dd540fabb1acfe5a3c641f5: Neither blocks nor log segments found. Creating new log.
I20260929 03:17:33.761072 21281 log.cc:824] T 00000000000000000000000000000000 P 76abf8500dd540fabb1acfe5a3c641f5: Log is configured to *not* fsync() on all Append() calls
I20260929 03:17:33.761711 21281 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76abf8500dd540fabb1acfe5a3c641f5: No bootstrap required, opened a new log
I20260929 03:17:33.763157 21281 raft_consensus.cc:348] T 00000000000000000000000000000000 P 76abf8500dd540fabb1acfe5a3c641f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76abf8500dd540fabb1acfe5a3c641f5" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 33269 } }
I20260929 03:17:33.763285 21281 raft_consensus.cc:374] T 00000000000000000000000000000000 P 76abf8500dd540fabb1acfe5a3c641f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:17:33.763306 21281 raft_consensus.cc:729] T 00000000000000000000000000000000 P 76abf8500dd540fabb1acfe5a3c641f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76abf8500dd540fabb1acfe5a3c641f5, State: Initialized, Role: FOLLOWER
I20260929 03:17:33.763437 21281 consensus_queue.cc:260] T 00000000000000000000000000000000 P 76abf8500dd540fabb1acfe5a3c641f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76abf8500dd540fabb1acfe5a3c641f5" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 33269 } }
I20260929 03:17:33.763516 21281 raft_consensus.cc:388] T 00000000000000000000000000000000 P 76abf8500dd540fabb1acfe5a3c641f5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:17:33.763538 21281 raft_consensus.cc:482] T 00000000000000000000000000000000 P 76abf8500dd540fabb1acfe5a3c641f5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:17:33.763566 21281 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 76abf8500dd540fabb1acfe5a3c641f5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:17:33.764026 21281 raft_consensus.cc:504] T 00000000000000000000000000000000 P 76abf8500dd540fabb1acfe5a3c641f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76abf8500dd540fabb1acfe5a3c641f5" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 33269 } }
I20260929 03:17:33.764106 21281 leader_election.cc:302] T 00000000000000000000000000000000 P 76abf8500dd540fabb1acfe5a3c641f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 76abf8500dd540fabb1acfe5a3c641f5; no voters:
I20260929 03:17:33.764228 21281 leader_election.cc:288] T 00000000000000000000000000000000 P 76abf8500dd540fabb1acfe5a3c641f5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:17:33.764308 21284 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 76abf8500dd540fabb1acfe5a3c641f5 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:17:33.764463 21284 raft_consensus.cc:686] T 00000000000000000000000000000000 P 76abf8500dd540fabb1acfe5a3c641f5 [term 1 LEADER]: Becoming Leader. State: Replica: 76abf8500dd540fabb1acfe5a3c641f5, State: Running, Role: LEADER
I20260929 03:17:33.764477 21281 sys_catalog.cc:564] T 00000000000000000000000000000000 P 76abf8500dd540fabb1acfe5a3c641f5 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:17:33.764557 21284 consensus_queue.cc:237] T 00000000000000000000000000000000 P 76abf8500dd540fabb1acfe5a3c641f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76abf8500dd540fabb1acfe5a3c641f5" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 33269 } }
I20260929 03:17:33.764588 21193 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuoaHbX/build/release/bin/kudu as pid 21213
I20260929 03:17:33.764750 21193 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651852476208-21193-0/minicluster-data/master-0/wal/instance
I20260929 03:17:33.764936 21286 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76abf8500dd540fabb1acfe5a3c641f5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 76abf8500dd540fabb1acfe5a3c641f5. Latest consensus state: current_term: 1 leader_uuid: "76abf8500dd540fabb1acfe5a3c641f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76abf8500dd540fabb1acfe5a3c641f5" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 33269 } } }
I20260929 03:17:33.765074 21285 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76abf8500dd540fabb1acfe5a3c641f5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "76abf8500dd540fabb1acfe5a3c641f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76abf8500dd540fabb1acfe5a3c641f5" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 33269 } } }
I20260929 03:17:33.765142 21285 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76abf8500dd540fabb1acfe5a3c641f5 [sys.catalog]: This master's current role is: LEADER
I20260929 03:17:33.765069 21286 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76abf8500dd540fabb1acfe5a3c641f5 [sys.catalog]: This master's current role is: LEADER
I20260929 03:17:33.765558 21193 external_mini_cluster.cc:1428] Setting key 37485315d2cc9b63fd852a2e36821762
I20260929 03:17:33.765950 21293 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:17:33.767149 21293 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:17:33.769316 21293 catalog_manager.cc:1347] Generated new cluster ID: 67abb425f7e846e787f1e6aaaed13ea5
I20260929 03:17:33.769366 21293 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 29 03:17:33 dist-test-slave-c9qh krb5kdc[21200](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651853, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.178.126@KRBTEST.COM
I20260929 03:17:33.781006 21301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:33.766445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37180 (local address 127.20.178.126:33269)
0929 03:17:33.766626 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:33.766635 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:33.766879 (+ 244us) server_negotiation.cc:407] Connection header received
0929 03:17:33.769188 (+ 2309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:33.769197 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:33.769255 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:33.769364 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:17:33.771703 (+ 2339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:33.772711 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:33.773687 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:33.774096 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:33.774249 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:33.777528 (+ 3279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:17:33.777533 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:17:33.777533 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:17:33.777551 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:17:33.779191 (+ 1640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:33.779612 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:33.779615 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:33.779615 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:33.779707 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:33.779956 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:33.779957 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:33.779958 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:33.780185 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:17:33.780330 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0929 03:17:33.780601 (+ 271us) server_negotiation.cc:299] Negotiation successful
0929 03:17:33.780763 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":116,"thread_start_us":82,"threads_started":1}
I20260929 03:17:33.787956 21293 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:17:33.788666 21293 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:17:33.792438 21293 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 76abf8500dd540fabb1acfe5a3c641f5: Generated new TSK 0
I20260929 03:17:33.792662 21293 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.178.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.178.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.178.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.65.keytab.
Entry for principal kudu/127.20.178.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.65.keytab.
I20260929 03:17:33.823091 21193 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuoaHbX/build/release/bin/kudu
/tmp/dist-test-taskuoaHbX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651852476208-21193-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651852476208-21193-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651852476208-21193-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651852476208-21193-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.178.65:0
--local_ip_for_outbound_sockets=127.20.178.65
--webserver_interface=127.20.178.65
--webserver_port=0
--tserver_master_addrs=127.20.178.126:33269
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.65.keytab
--principal=kudu/127.20.178.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.178.84:42513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651852476208-21193-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651852476208-21193-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651852476208-21193-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:17:33.925202 21307 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:17:33.925395 21307 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:17:33.925418 21307 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:17:33.926568 21307 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:17:33.926609 21307 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:17:33.926637 21307 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.178.65
I20260929 03:17:33.928059 21307 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.178.84:42513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651852476208-21193-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651852476208-21193-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.65.keytab
--principal=kudu/127.20.178.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.178.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651852476208-21193-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.178.65
--webserver_port=0
--tserver_master_addrs=127.20.178.126:33269
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21307
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.178.65
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651852476208-21193-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:17:33.928304 21307 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:17:33.928546 21307 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 03:17:33.933539 21307 server_base.cc:1034] running on GCE node
W20260929 03:17:33.933511 21313 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:17:33.933506 21315 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:17:33.933522 21312 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:17:33.933935 21307 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:17:33.934190 21307 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:17:33.935359 21307 hybrid_clock.cc:648] HybridClock initialized: now 1790651853935356 us; error 37 us; skew 500 ppm
Sep 29 03:17:33 dist-test-slave-c9qh krb5kdc[21200](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651853, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:17:33.937925 21307 init.cc:377] Logged in from keytab as kudu/127.20.178.65@KRBTEST.COM (short username kudu)
I20260929 03:17:33.938347 21307 webserver.cc:460] Webserver started at http://127.20.178.65:37409/ using document root <none> and password file <none>
I20260929 03:17:33.938570 21307 fs_manager.cc:359] Metadata directory not provided
I20260929 03:17:33.938637 21307 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:17:33.938802 21307 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:17:33.939934 21307 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651852476208-21193-0/minicluster-data/ts-0/data/instance:
uuid: "24c4838bb71a4407b7fb5b7f6026fa82"
format_stamp: "Formatted at 2026-09-29 03:17:33 on dist-test-slave-c9qh"
server_key: "9f8b15af1190080e263d9b76a829146b"
server_key_iv: "6f47564f0b69ed9b857202475e4d272d"
server_key_version: "encryptionkey@0"
I20260929 03:17:33.940282 21307 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651852476208-21193-0/minicluster-data/ts-0/wal/instance:
uuid: "24c4838bb71a4407b7fb5b7f6026fa82"
format_stamp: "Formatted at 2026-09-29 03:17:33 on dist-test-slave-c9qh"
server_key: "9f8b15af1190080e263d9b76a829146b"
server_key_iv: "6f47564f0b69ed9b857202475e4d272d"
server_key_version: "encryptionkey@0"
I20260929 03:17:33.941684 21307 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 03:17:33.942427 21322 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:17:33.942668 21307 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260929 03:17:33.942767 21307 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651852476208-21193-0/minicluster-data/ts-0/data,/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651852476208-21193-0/minicluster-data/ts-0/wal
uuid: "24c4838bb71a4407b7fb5b7f6026fa82"
format_stamp: "Formatted at 2026-09-29 03:17:33 on dist-test-slave-c9qh"
server_key: "9f8b15af1190080e263d9b76a829146b"
server_key_iv: "6f47564f0b69ed9b857202475e4d272d"
server_key_version: "encryptionkey@0"
I20260929 03:17:33.942852 21307 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651852476208-21193-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651852476208-21193-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651852476208-21193-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:17:33.959388 21307 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:17:33.961200 21307 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:17:33.961361 21307 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:17:33.961583 21307 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:17:33.961901 21307 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:17:33.961956 21307 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:17:33.961997 21307 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:17:33.962024 21307 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:17:33.967862 21307 rpc_server.cc:307] RPC server started. Bound to: 127.20.178.65:37899
I20260929 03:17:33.967937 21435 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.178.65:37899 every 8 connection(s)
I20260929 03:17:33.968250 21307 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651852476208-21193-0/minicluster-data/ts-0/data/info.pb
I20260929 03:17:33.968485 21193 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuoaHbX/build/release/bin/kudu as pid 21307
I20260929 03:17:33.968573 21193 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651852476208-21193-0/minicluster-data/ts-0/wal/instance
I20260929 03:17:33.968766 21193 external_mini_cluster.cc:1428] Setting key b5a13f853bba22240c17b15c82033e41
Sep 29 03:17:33 dist-test-slave-c9qh krb5kdc[21200](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651853, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.65@KRBTEST.COM for kudu/127.20.178.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.20.178.66@KRBTEST.COM; defaulting to no policy
I20260929 03:17:33.978016 21301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:33.969052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.65:38725 (local address 127.20.178.126:33269)
0929 03:17:33.969124 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:33.969127 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:33.969339 (+ 212us) server_negotiation.cc:407] Connection header received
0929 03:17:33.969803 (+ 464us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:33.969804 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:33.969821 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:33.969891 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:17:33.970748 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:33.971551 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:33.972574 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:33.972854 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:33.972941 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:33.975089 (+ 2148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:17:33.975090 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:17:33.975090 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:17:33.975097 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:17:33.976591 (+ 1494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:33.977032 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:33.977033 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:33.977034 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:33.977098 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:33.977350 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:33.977352 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:33.977352 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:33.977471 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:17:33.977548 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0929 03:17:33.977830 (+ 282us) server_negotiation.cc:299] Negotiation successful
0929 03:17:33.977925 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 03:17:33.978220 21438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:33.969029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:33269 (local address 127.20.178.65:38725)
0929 03:17:33.969202 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0929 03:17:33.969246 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 03:17:33.969709 (+ 463us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:17:33.969940 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:17:33.969943 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:17:33.970493 (+ 550us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:17:33.970675 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:33.970678 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:33.971644 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:33.971645 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:33.972458 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:33.972460 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:33.972951 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:33.972952 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:33.973105 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:33.973631 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:17:33.973641 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:17:33.974974 (+ 1333us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:17:33.976712 (+ 1738us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:17:33.976719 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:17:33.976721 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:17:33.976940 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:17:33.977164 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:17:33.977165 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:17:33.977165 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:17:33.977280 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:17:33.977540 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:17:33.977547 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:17:33.977686 (+ 139us) client_negotiation.cc:769] Sending connection context
0929 03:17:33.977796 (+ 110us) client_negotiation.cc:240] Negotiation successful
0929 03:17:33.977977 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":93,"threads_started":1}
I20260929 03:17:33.978714 21436 heartbeater.cc:347] Connected to a master server at 127.20.178.126:33269
I20260929 03:17:33.978792 21436 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.178.66@KRBTEST.COM" created.
I20260929 03:17:33.978986 21436 heartbeater.cc:511] Master 127.20.178.126:33269 requested a full tablet report, sending...
I20260929 03:17:33.979537 21245 ts_manager.cc:194] Registered new tserver with Master: 24c4838bb71a4407b7fb5b7f6026fa82 (127.20.178.65:37899)
I20260929 03:17:33.980155 21245 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.178.65@KRBTEST.COM'} at 127.20.178.65:38725
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.178.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.66.keytab.
Entry for principal kudu/127.20.178.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.66.keytab.
I20260929 03:17:33.993352 21193 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuoaHbX/build/release/bin/kudu
/tmp/dist-test-taskuoaHbX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651852476208-21193-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651852476208-21193-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651852476208-21193-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651852476208-21193-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.178.66:0
--local_ip_for_outbound_sockets=127.20.178.66
--webserver_interface=127.20.178.66
--webserver_port=0
--tserver_master_addrs=127.20.178.126:33269
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.66.keytab
--principal=kudu/127.20.178.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.178.84:42513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651852476208-21193-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651852476208-21193-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651852476208-21193-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:17:34.073180 21441 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:17:34.073369 21441 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:17:34.073391 21441 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:17:34.074440 21441 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:17:34.074473 21441 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:17:34.074499 21441 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.178.66
I20260929 03:17:34.075846 21441 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.178.84:42513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651852476208-21193-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651852476208-21193-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.66.keytab
--principal=kudu/127.20.178.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.178.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651852476208-21193-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.178.66
--webserver_port=0
--tserver_master_addrs=127.20.178.126:33269
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21441
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.178.66
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651852476208-21193-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:17:34.076030 21441 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:17:34.076215 21441 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:17:34.081276 21446 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:17:34.081434 21449 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:17:34.081442 21447 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:17:34.081513 21441 server_base.cc:1034] running on GCE node
I20260929 03:17:34.081700 21441 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:17:34.081956 21441 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:17:34.083127 21441 hybrid_clock.cc:648] HybridClock initialized: now 1790651854083133 us; error 48 us; skew 500 ppm
Sep 29 03:17:34 dist-test-slave-c9qh krb5kdc[21200](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651854, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:17:34.085858 21441 init.cc:377] Logged in from keytab as kudu/127.20.178.66@KRBTEST.COM (short username kudu)
I20260929 03:17:34.086285 21441 webserver.cc:460] Webserver started at http://127.20.178.66:44745/ using document root <none> and password file <none>
I20260929 03:17:34.086515 21441 fs_manager.cc:359] Metadata directory not provided
I20260929 03:17:34.086586 21441 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:17:34.086738 21441 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:17:34.087775 21441 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651852476208-21193-0/minicluster-data/ts-1/data/instance:
uuid: "8266f5596a27452fb764828b1c0618e2"
format_stamp: "Formatted at 2026-09-29 03:17:34 on dist-test-slave-c9qh"
server_key: "57c9d5a2bef204566bc545665b30ce83"
server_key_iv: "b2c0fc5725d75f0aad276f793875480e"
server_key_version: "encryptionkey@0"
I20260929 03:17:34.088132 21441 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651852476208-21193-0/minicluster-data/ts-1/wal/instance:
uuid: "8266f5596a27452fb764828b1c0618e2"
format_stamp: "Formatted at 2026-09-29 03:17:34 on dist-test-slave-c9qh"
server_key: "57c9d5a2bef204566bc545665b30ce83"
server_key_iv: "b2c0fc5725d75f0aad276f793875480e"
server_key_version: "encryptionkey@0"
I20260929 03:17:34.089535 21441 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 03:17:34.090282 21456 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:17:34.090451 21441 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260929 03:17:34.090545 21441 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651852476208-21193-0/minicluster-data/ts-1/data,/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651852476208-21193-0/minicluster-data/ts-1/wal
uuid: "8266f5596a27452fb764828b1c0618e2"
format_stamp: "Formatted at 2026-09-29 03:17:34 on dist-test-slave-c9qh"
server_key: "57c9d5a2bef204566bc545665b30ce83"
server_key_iv: "b2c0fc5725d75f0aad276f793875480e"
server_key_version: "encryptionkey@0"
I20260929 03:17:34.090623 21441 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651852476208-21193-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651852476208-21193-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651852476208-21193-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:17:34.104563 21441 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:17:34.106443 21441 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:17:34.106619 21441 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:17:34.106932 21441 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:17:34.107309 21441 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:17:34.107359 21441 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:17:34.107393 21441 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:17:34.107419 21441 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:17:34.113058 21441 rpc_server.cc:307] RPC server started. Bound to: 127.20.178.66:42721
I20260929 03:17:34.113091 21569 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.178.66:42721 every 8 connection(s)
I20260929 03:17:34.113448 21441 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651852476208-21193-0/minicluster-data/ts-1/data/info.pb
I20260929 03:17:34.117851 21193 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuoaHbX/build/release/bin/kudu as pid 21441
I20260929 03:17:34.117924 21193 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651852476208-21193-0/minicluster-data/ts-1/wal/instance
I20260929 03:17:34.118085 21193 external_mini_cluster.cc:1428] Setting key 7de3ff8894d82e7c41ef6f4c711ae4a9
Sep 29 03:17:34 dist-test-slave-c9qh krb5kdc[21200](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651854, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.66@KRBTEST.COM for kudu/127.20.178.126@KRBTEST.COM
I20260929 03:17:34.123202 21572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:34.114191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:33269 (local address 127.20.178.66:38031)
0929 03:17:34.114358 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0929 03:17:34.114392 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0929 03:17:34.114855 (+ 463us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:17:34.115054 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:17:34.115058 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:17:34.115611 (+ 553us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:17:34.115794 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:34.115798 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:34.116454 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:34.116455 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:34.117287 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:34.117290 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:34.117634 (+ 344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:34.117634 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:34.117758 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:34.118313 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:17:34.118320 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:17:34.119728 (+ 1408us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:17:34.121730 (+ 2002us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:17:34.121735 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:17:34.121737 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:17:34.121940 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:17:34.122193 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:17:34.122194 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:17:34.122195 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:17:34.122313 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:17:34.122719 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:17:34.122725 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:17:34.122836 (+ 111us) client_negotiation.cc:769] Sending connection context
0929 03:17:34.122929 (+ 93us) client_negotiation.cc:240] Negotiation successful
0929 03:17:34.123043 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":85,"threads_started":1}
I20260929 03:17:34.123211 21301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:34.114209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.66:38031 (local address 127.20.178.126:33269)
0929 03:17:34.114280 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:34.114282 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:34.114480 (+ 198us) server_negotiation.cc:407] Connection header received
0929 03:17:34.114938 (+ 458us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:34.114939 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:34.114949 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:34.114996 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:17:34.115882 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:34.116375 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:34.117367 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:34.117558 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:34.117617 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:34.119811 (+ 2194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:17:34.119812 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:17:34.119813 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:17:34.119821 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:17:34.121654 (+ 1833us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:34.122020 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:34.122021 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:34.122021 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:34.122086 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:34.122433 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:34.122434 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:34.122434 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:34.122627 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:17:34.122707 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0929 03:17:34.122972 (+ 265us) server_negotiation.cc:299] Negotiation successful
0929 03:17:34.123076 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260929 03:17:34.123761 21570 heartbeater.cc:347] Connected to a master server at 127.20.178.126:33269
I20260929 03:17:34.123855 21570 heartbeater.cc:464] Registering TS with master...
I20260929 03:17:34.124051 21570 heartbeater.cc:511] Master 127.20.178.126:33269 requested a full tablet report, sending...
I20260929 03:17:34.124469 21245 ts_manager.cc:194] Registered new tserver with Master: 8266f5596a27452fb764828b1c0618e2 (127.20.178.66:42721)
I20260929 03:17:34.124887 21245 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.178.66@KRBTEST.COM'} at 127.20.178.66:38031
WARNING: no policy specified for kudu/127.20.178.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.178.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.178.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.67.keytab.
Entry for principal kudu/127.20.178.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.67.keytab.
I20260929 03:17:34.143672 21193 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuoaHbX/build/release/bin/kudu
/tmp/dist-test-taskuoaHbX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651852476208-21193-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651852476208-21193-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651852476208-21193-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651852476208-21193-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.178.67:0
--local_ip_for_outbound_sockets=127.20.178.67
--webserver_interface=127.20.178.67
--webserver_port=0
--tserver_master_addrs=127.20.178.126:33269
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.67.keytab
--principal=kudu/127.20.178.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.178.84:42513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651852476208-21193-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651852476208-21193-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651852476208-21193-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:17:34.223042 21575 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:17:34.223265 21575 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:17:34.223300 21575 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:17:34.224413 21575 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:17:34.224481 21575 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:17:34.224524 21575 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.178.67
I20260929 03:17:34.225970 21575 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.178.84:42513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651852476208-21193-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651852476208-21193-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.67.keytab
--principal=kudu/127.20.178.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.178.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651852476208-21193-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.178.67
--webserver_port=0
--tserver_master_addrs=127.20.178.126:33269
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21575
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.178.67
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651852476208-21193-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:17:34.226225 21575 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:17:34.226445 21575 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 03:17:34.231514 21575 server_base.cc:1034] running on GCE node
W20260929 03:17:34.231597 21583 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:17:34.231398 21580 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:17:34.231570 21581 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:17:34.231962 21575 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:17:34.232174 21575 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:17:34.233309 21575 hybrid_clock.cc:648] HybridClock initialized: now 1790651854233290 us; error 36 us; skew 500 ppm
Sep 29 03:17:34 dist-test-slave-c9qh krb5kdc[21200](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651854, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:17:34.235705 21575 init.cc:377] Logged in from keytab as kudu/127.20.178.67@KRBTEST.COM (short username kudu)
I20260929 03:17:34.236166 21575 webserver.cc:460] Webserver started at http://127.20.178.67:36557/ using document root <none> and password file <none>
I20260929 03:17:34.236382 21575 fs_manager.cc:359] Metadata directory not provided
I20260929 03:17:34.236447 21575 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:17:34.236625 21575 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:17:34.237672 21575 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651852476208-21193-0/minicluster-data/ts-2/data/instance:
uuid: "809eacd99d5e41c092a519fd3906dae5"
format_stamp: "Formatted at 2026-09-29 03:17:34 on dist-test-slave-c9qh"
server_key: "13591448166ed5295f6537ac62840031"
server_key_iv: "3539031f323b5bebf629f6d5d9e09345"
server_key_version: "encryptionkey@0"
I20260929 03:17:34.238021 21575 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651852476208-21193-0/minicluster-data/ts-2/wal/instance:
uuid: "809eacd99d5e41c092a519fd3906dae5"
format_stamp: "Formatted at 2026-09-29 03:17:34 on dist-test-slave-c9qh"
server_key: "13591448166ed5295f6537ac62840031"
server_key_iv: "3539031f323b5bebf629f6d5d9e09345"
server_key_version: "encryptionkey@0"
I20260929 03:17:34.239379 21575 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 03:17:34.240161 21590 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:17:34.240356 21575 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 03:17:34.240466 21575 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651852476208-21193-0/minicluster-data/ts-2/data,/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651852476208-21193-0/minicluster-data/ts-2/wal
uuid: "809eacd99d5e41c092a519fd3906dae5"
format_stamp: "Formatted at 2026-09-29 03:17:34 on dist-test-slave-c9qh"
server_key: "13591448166ed5295f6537ac62840031"
server_key_iv: "3539031f323b5bebf629f6d5d9e09345"
server_key_version: "encryptionkey@0"
I20260929 03:17:34.240533 21575 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651852476208-21193-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651852476208-21193-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651852476208-21193-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:17:34.249094 21575 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:17:34.250801 21575 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:17:34.250903 21575 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:17:34.251067 21575 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:17:34.251368 21575 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:17:34.251400 21575 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:17:34.251442 21575 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:17:34.251470 21575 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:17:34.256955 21575 rpc_server.cc:307] RPC server started. Bound to: 127.20.178.67:37741
I20260929 03:17:34.257010 21703 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.178.67:37741 every 8 connection(s)
I20260929 03:17:34.257323 21575 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651852476208-21193-0/minicluster-data/ts-2/data/info.pb
I20260929 03:17:34.258278 21193 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuoaHbX/build/release/bin/kudu as pid 21575
I20260929 03:17:34.258375 21193 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651852476208-21193-0/minicluster-data/ts-2/wal/instance
I20260929 03:17:34.258556 21193 external_mini_cluster.cc:1428] Setting key 39733e623c44ff03754f1d8648ae2a1b
Sep 29 03:17:34 dist-test-slave-c9qh krb5kdc[21200](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651854, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.67@KRBTEST.COM for kudu/127.20.178.126@KRBTEST.COM
I20260929 03:17:34.266788 21301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:34.258020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.67:42193 (local address 127.20.178.126:33269)
0929 03:17:34.258104 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:34.258107 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:34.258284 (+ 177us) server_negotiation.cc:407] Connection header received
0929 03:17:34.258945 (+ 661us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:34.258947 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:34.258957 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:34.259014 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:17:34.260163 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:34.260618 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:34.261694 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:34.261872 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:34.261922 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:34.263928 (+ 2006us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:17:34.263929 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:17:34.263929 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:17:34.263935 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:17:34.265580 (+ 1645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:34.265932 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:34.265933 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:34.265933 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:34.265984 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:34.266297 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:34.266298 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:34.266298 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:34.266398 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:17:34.266448 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0929 03:17:34.266670 (+ 222us) server_negotiation.cc:299] Negotiation successful
0929 03:17:34.266727 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260929 03:17:34.266966 21706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:34.257982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:33269 (local address 127.20.178.67:42193)
0929 03:17:34.258167 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0929 03:17:34.258201 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0929 03:17:34.258840 (+ 639us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:17:34.259095 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:17:34.259099 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:17:34.259902 (+ 803us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:17:34.260077 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:34.260082 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:34.260764 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:34.260765 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:34.261620 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:34.261622 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:34.261936 (+ 314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:34.261937 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:34.262030 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:34.262518 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:17:34.262525 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:17:34.263844 (+ 1319us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:17:34.265653 (+ 1809us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:17:34.265658 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:17:34.265660 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:17:34.265871 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:17:34.266089 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:17:34.266090 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:17:34.266090 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:17:34.266232 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:17:34.266453 (+ 221us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:17:34.266457 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:17:34.266563 (+ 106us) client_negotiation.cc:769] Sending connection context
0929 03:17:34.266658 (+ 95us) client_negotiation.cc:240] Negotiation successful
0929 03:17:34.266768 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":89,"threads_started":1}
I20260929 03:17:34.267571 21704 heartbeater.cc:347] Connected to a master server at 127.20.178.126:33269
I20260929 03:17:34.267675 21704 heartbeater.cc:464] Registering TS with master...
I20260929 03:17:34.267838 21704 heartbeater.cc:511] Master 127.20.178.126:33269 requested a full tablet report, sending...
I20260929 03:17:34.268219 21245 ts_manager.cc:194] Registered new tserver with Master: 809eacd99d5e41c092a519fd3906dae5 (127.20.178.67:37741)
I20260929 03:17:34.268625 21245 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.178.67@KRBTEST.COM'} at 127.20.178.67:42193
I20260929 03:17:34.270581 21193 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 03:17:34 dist-test-slave-c9qh krb5kdc[21200](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651854, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 03:17:34 dist-test-slave-c9qh krb5kdc[21200](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651854, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.20.178.65@KRBTEST.COM
I20260929 03:17:34.294458 21711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:34.286443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39650 (local address 127.20.178.65:37899)
0929 03:17:34.286668 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:34.286673 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:34.286683 (+ 10us) server_negotiation.cc:407] Connection header received
0929 03:17:34.286707 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:34.286708 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:34.286722 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:34.286787 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:17:34.287301 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:34.287992 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:34.288815 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:34.289151 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:34.289208 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:34.291522 (+ 2314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:17:34.291531 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:17:34.291531 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:17:34.291546 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:17:34.293186 (+ 1640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:34.293622 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:34.293626 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:34.293626 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:34.293690 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:34.293891 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:34.293892 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:34.293893 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:34.294031 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:17:34.294204 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0929 03:17:34.294259 (+ 55us) server_negotiation.cc:299] Negotiation successful
0929 03:17:34.294339 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":140,"threads_started":1}
W20260929 03:17:34.294881 21350 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39650
Sep 29 03:17:34 dist-test-slave-c9qh krb5kdc[21200](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651854, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 03:17:34 dist-test-slave-c9qh krb5kdc[21200](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651854, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.178.65@KRBTEST.COM
I20260929 03:17:34.315673 21711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:34.309214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39658 (local address 127.20.178.65:37899)
0929 03:17:34.309286 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:34.309288 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:34.309298 (+ 10us) server_negotiation.cc:407] Connection header received
0929 03:17:34.309366 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:34.309366 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:34.309375 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:34.309423 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:17:34.309957 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:34.310428 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:34.311106 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:34.311297 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:34.311352 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:34.313478 (+ 2126us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:17:34.313479 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:17:34.313479 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:17:34.313486 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:17:34.314762 (+ 1276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:34.315049 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:34.315049 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:34.315050 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:34.315100 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:34.315242 (+ 142us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:34.315243 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:34.315243 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:34.315320 (+ 77us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:17:34.315397 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0929 03:17:34.315465 (+ 68us) server_negotiation.cc:299] Negotiation successful
0929 03:17:34.315524 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 03:17:34.316268 21193 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuoaHbX/build/release/bin/kudu with pid 21307
I20260929 03:17:34.320842 21193 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuoaHbX/build/release/bin/kudu with pid 21441
I20260929 03:17:34.326229 21193 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuoaHbX/build/release/bin/kudu with pid 21575
I20260929 03:17:34.330276 21193 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuoaHbX/build/release/bin/kudu with pid 21213
2026-09-29T03:17:34Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (1861 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651852476208-21193-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:17:34 dist-test-slave-c9qh krb5kdc[21718](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:17:34 dist-test-slave-c9qh krb5kdc[21718](info): set up 2 sockets
Sep 29 03:17:34 dist-test-slave-c9qh krb5kdc[21718](info): commencing operation
krb5kdc: starting...
W20260929 03:17:35.384162 21193 mini_kdc.cc:121] Time spent starting KDC: real 1.034s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651852476208-21193-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651852476208-21193-0/krb5kdc/test-user.keytab.
Sep 29 03:17:35 dist-test-slave-c9qh krb5kdc[21718](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651855, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:17:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:17:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.178.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.178.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.178.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.126.keytab.
Entry for principal kudu/127.20.178.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.126.keytab.
I20260929 03:17:35.497598 21193 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuoaHbX/build/release/bin/kudu
/tmp/dist-test-taskuoaHbX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651852476208-21193-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651852476208-21193-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651852476208-21193-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651852476208-21193-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.178.126:35317
--webserver_interface=127.20.178.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.126.keytab
--principal=kudu/127.20.178.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.178.84:46241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.178.126:35317
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651852476208-21193-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651852476208-21193-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651852476208-21193-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:17:35.576674 21731 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:17:35.576869 21731 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:17:35.576891 21731 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:17:35.578068 21731 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:17:35.578112 21731 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:17:35.578127 21731 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:17:35.578140 21731 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:17:35.578151 21731 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:17:35.579630 21731 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.178.84:46241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651852476208-21193-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651852476208-21193-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.178.126:35317
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.126.keytab
--principal=kudu/127.20.178.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.178.126:35317
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651852476208-21193-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.178.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21731
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651852476208-21193-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:17:35.579906 21731 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:17:35.580143 21731 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:17:35.585180 21739 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:17:35.585192 21737 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:17:35.585194 21736 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:17:35.585388 21731 server_base.cc:1034] running on GCE node
I20260929 03:17:35.585654 21731 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:17:35.585942 21731 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:17:35.587108 21731 hybrid_clock.cc:648] HybridClock initialized: now 1790651855587073 us; error 47 us; skew 500 ppm
Sep 29 03:17:35 dist-test-slave-c9qh krb5kdc[21718](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651855, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:17:35.589586 21731 init.cc:377] Logged in from keytab as kudu/127.20.178.126@KRBTEST.COM (short username kudu)
I20260929 03:17:35.590086 21731 webserver.cc:460] Webserver started at http://127.20.178.126:35813/ using document root <none> and password file <none>
I20260929 03:17:35.590334 21731 fs_manager.cc:359] Metadata directory not provided
I20260929 03:17:35.590404 21731 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:17:35.590551 21731 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:17:35.591665 21731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651852476208-21193-0/minicluster-data/master-0/data/instance:
uuid: "168066af915d4f9dbc4750201267bfa7"
format_stamp: "Formatted at 2026-09-29 03:17:35 on dist-test-slave-c9qh"
server_key: "3209ff2b5b115df7ad26b08fe490147f"
server_key_iv: "c544876b05c9a9294628a9d278048f62"
server_key_version: "encryptionkey@0"
I20260929 03:17:35.592010 21731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651852476208-21193-0/minicluster-data/master-0/wal/instance:
uuid: "168066af915d4f9dbc4750201267bfa7"
format_stamp: "Formatted at 2026-09-29 03:17:35 on dist-test-slave-c9qh"
server_key: "3209ff2b5b115df7ad26b08fe490147f"
server_key_iv: "c544876b05c9a9294628a9d278048f62"
server_key_version: "encryptionkey@0"
I20260929 03:17:35.593397 21731 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 03:17:35.594122 21746 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:17:35.594282 21731 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 03:17:35.594386 21731 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651852476208-21193-0/minicluster-data/master-0/data,/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651852476208-21193-0/minicluster-data/master-0/wal
uuid: "168066af915d4f9dbc4750201267bfa7"
format_stamp: "Formatted at 2026-09-29 03:17:35 on dist-test-slave-c9qh"
server_key: "3209ff2b5b115df7ad26b08fe490147f"
server_key_iv: "c544876b05c9a9294628a9d278048f62"
server_key_version: "encryptionkey@0"
I20260929 03:17:35.594460 21731 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651852476208-21193-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651852476208-21193-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651852476208-21193-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:17:35.608053 21731 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:17:35.609953 21731 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:17:35.610116 21731 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:17:35.614249 21731 rpc_server.cc:307] RPC server started. Bound to: 127.20.178.126:35317
I20260929 03:17:35.614287 21798 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.178.126:35317 every 8 connection(s)
I20260929 03:17:35.614642 21731 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651852476208-21193-0/minicluster-data/master-0/data/info.pb
I20260929 03:17:35.615481 21799 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:17:35.617810 21799 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 168066af915d4f9dbc4750201267bfa7: Bootstrap starting.
I20260929 03:17:35.618330 21799 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 168066af915d4f9dbc4750201267bfa7: Neither blocks nor log segments found. Creating new log.
I20260929 03:17:35.618557 21799 log.cc:824] T 00000000000000000000000000000000 P 168066af915d4f9dbc4750201267bfa7: Log is configured to *not* fsync() on all Append() calls
I20260929 03:17:35.619320 21799 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 168066af915d4f9dbc4750201267bfa7: No bootstrap required, opened a new log
I20260929 03:17:35.620855 21799 raft_consensus.cc:348] T 00000000000000000000000000000000 P 168066af915d4f9dbc4750201267bfa7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "168066af915d4f9dbc4750201267bfa7" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 35317 } }
I20260929 03:17:35.620992 21799 raft_consensus.cc:374] T 00000000000000000000000000000000 P 168066af915d4f9dbc4750201267bfa7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:17:35.621014 21799 raft_consensus.cc:729] T 00000000000000000000000000000000 P 168066af915d4f9dbc4750201267bfa7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 168066af915d4f9dbc4750201267bfa7, State: Initialized, Role: FOLLOWER
I20260929 03:17:35.621160 21799 consensus_queue.cc:260] T 00000000000000000000000000000000 P 168066af915d4f9dbc4750201267bfa7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "168066af915d4f9dbc4750201267bfa7" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 35317 } }
I20260929 03:17:35.621246 21799 raft_consensus.cc:388] T 00000000000000000000000000000000 P 168066af915d4f9dbc4750201267bfa7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:17:35.621270 21799 raft_consensus.cc:482] T 00000000000000000000000000000000 P 168066af915d4f9dbc4750201267bfa7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:17:35.621299 21799 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 168066af915d4f9dbc4750201267bfa7 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:17:35.621779 21799 raft_consensus.cc:504] T 00000000000000000000000000000000 P 168066af915d4f9dbc4750201267bfa7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "168066af915d4f9dbc4750201267bfa7" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 35317 } }
I20260929 03:17:35.621863 21799 leader_election.cc:302] T 00000000000000000000000000000000 P 168066af915d4f9dbc4750201267bfa7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 168066af915d4f9dbc4750201267bfa7; no voters:
I20260929 03:17:35.621914 21193 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuoaHbX/build/release/bin/kudu as pid 21731
I20260929 03:17:35.622002 21193 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651852476208-21193-0/minicluster-data/master-0/wal/instance
I20260929 03:17:35.622051 21799 leader_election.cc:288] T 00000000000000000000000000000000 P 168066af915d4f9dbc4750201267bfa7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:17:35.622182 21193 external_mini_cluster.cc:1428] Setting key 1823d501713b77dd870c9aa5ceba3e55
I20260929 03:17:35.622174 21802 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 168066af915d4f9dbc4750201267bfa7 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:17:35.622320 21799 sys_catalog.cc:564] T 00000000000000000000000000000000 P 168066af915d4f9dbc4750201267bfa7 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:17:35.622392 21802 raft_consensus.cc:686] T 00000000000000000000000000000000 P 168066af915d4f9dbc4750201267bfa7 [term 1 LEADER]: Becoming Leader. State: Replica: 168066af915d4f9dbc4750201267bfa7, State: Running, Role: LEADER
I20260929 03:17:35.622568 21802 consensus_queue.cc:237] T 00000000000000000000000000000000 P 168066af915d4f9dbc4750201267bfa7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "168066af915d4f9dbc4750201267bfa7" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 35317 } }
I20260929 03:17:35.623546 21803 sys_catalog.cc:455] T 00000000000000000000000000000000 P 168066af915d4f9dbc4750201267bfa7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "168066af915d4f9dbc4750201267bfa7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "168066af915d4f9dbc4750201267bfa7" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 35317 } } }
I20260929 03:17:35.623656 21803 sys_catalog.cc:458] T 00000000000000000000000000000000 P 168066af915d4f9dbc4750201267bfa7 [sys.catalog]: This master's current role is: LEADER
I20260929 03:17:35.623867 21814 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:17:35.624197 21804 sys_catalog.cc:455] T 00000000000000000000000000000000 P 168066af915d4f9dbc4750201267bfa7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 168066af915d4f9dbc4750201267bfa7. Latest consensus state: current_term: 1 leader_uuid: "168066af915d4f9dbc4750201267bfa7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "168066af915d4f9dbc4750201267bfa7" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 35317 } } }
I20260929 03:17:35.624277 21804 sys_catalog.cc:458] T 00000000000000000000000000000000 P 168066af915d4f9dbc4750201267bfa7 [sys.catalog]: This master's current role is: LEADER
I20260929 03:17:35.624595 21814 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:17:35.626067 21814 catalog_manager.cc:1347] Generated new cluster ID: df7468bd21e549f694529f6d0c93fd60
I20260929 03:17:35.626118 21814 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 29 03:17:35 dist-test-slave-c9qh krb5kdc[21718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651855, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.178.126@KRBTEST.COM
I20260929 03:17:35.637351 21808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:35.622803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34696 (local address 127.20.178.126:35317)
0929 03:17:35.623410 (+ 607us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:35.623421 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:35.623454 (+ 33us) server_negotiation.cc:407] Connection header received
0929 03:17:35.623632 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:35.623639 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:35.623705 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:35.623898 (+ 193us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:17:35.624673 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:35.626018 (+ 1345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:35.629675 (+ 3657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:35.630831 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:35.630952 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:35.633734 (+ 2782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:17:35.633739 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:17:35.633739 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:17:35.633751 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:17:35.635677 (+ 1926us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:35.636074 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:35.636077 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:35.636077 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:35.636185 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:35.636397 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:35.636398 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:35.636398 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:35.636726 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:17:35.636858 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0929 03:17:35.636985 (+ 127us) server_negotiation.cc:299] Negotiation successful
0929 03:17:35.637113 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":556,"thread_start_us":149,"threads_started":1}
I20260929 03:17:35.646970 21814 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:17:35.647725 21814 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:17:35.656399 21814 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 168066af915d4f9dbc4750201267bfa7: Generated new TSK 0
I20260929 03:17:35.656601 21814 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.178.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.178.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.178.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.65.keytab.
Entry for principal kudu/127.20.178.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.65.keytab.
I20260929 03:17:35.685595 21193 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuoaHbX/build/release/bin/kudu
/tmp/dist-test-taskuoaHbX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651852476208-21193-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651852476208-21193-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651852476208-21193-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651852476208-21193-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.178.65:0
--local_ip_for_outbound_sockets=127.20.178.65
--webserver_interface=127.20.178.65
--webserver_port=0
--tserver_master_addrs=127.20.178.126:35317
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.65.keytab
--principal=kudu/127.20.178.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.178.84:46241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651852476208-21193-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651852476208-21193-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651852476208-21193-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:17:35.766430 21825 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:17:35.766640 21825 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:17:35.766664 21825 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:17:35.767832 21825 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:17:35.767876 21825 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:17:35.767927 21825 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.178.65
I20260929 03:17:35.769685 21825 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.178.84:46241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651852476208-21193-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651852476208-21193-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.65.keytab
--principal=kudu/127.20.178.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.178.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651852476208-21193-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.178.65
--webserver_port=0
--tserver_master_addrs=127.20.178.126:35317
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21825
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.178.65
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651852476208-21193-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:17:35.769939 21825 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:17:35.770206 21825 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:17:35.775122 21830 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:17:35.775082 21831 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:17:35.775296 21825 server_base.cc:1034] running on GCE node
W20260929 03:17:35.775086 21833 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:17:35.775612 21825 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:17:35.775873 21825 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:17:35.777084 21825 hybrid_clock.cc:648] HybridClock initialized: now 1790651855777064 us; error 32 us; skew 500 ppm
Sep 29 03:17:35 dist-test-slave-c9qh krb5kdc[21718](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651855, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:17:35.780006 21825 init.cc:377] Logged in from keytab as kudu/127.20.178.65@KRBTEST.COM (short username kudu)
I20260929 03:17:35.780395 21825 webserver.cc:460] Webserver started at http://127.20.178.65:42703/ using document root <none> and password file <none>
I20260929 03:17:35.780570 21825 fs_manager.cc:359] Metadata directory not provided
I20260929 03:17:35.780608 21825 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:17:35.780781 21825 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:17:35.781774 21825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651852476208-21193-0/minicluster-data/ts-0/data/instance:
uuid: "6d3c1300047541b7b7d267ad536912e1"
format_stamp: "Formatted at 2026-09-29 03:17:35 on dist-test-slave-c9qh"
server_key: "475e9b01895039c0e631c4a62ff0e0de"
server_key_iv: "16602f170eff1cc55a4ccbe52211285b"
server_key_version: "encryptionkey@0"
I20260929 03:17:35.782063 21825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651852476208-21193-0/minicluster-data/ts-0/wal/instance:
uuid: "6d3c1300047541b7b7d267ad536912e1"
format_stamp: "Formatted at 2026-09-29 03:17:35 on dist-test-slave-c9qh"
server_key: "475e9b01895039c0e631c4a62ff0e0de"
server_key_iv: "16602f170eff1cc55a4ccbe52211285b"
server_key_version: "encryptionkey@0"
I20260929 03:17:35.785095 21825 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 03:17:35.786811 21840 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:17:35.787048 21825 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 03:17:35.787154 21825 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651852476208-21193-0/minicluster-data/ts-0/data,/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651852476208-21193-0/minicluster-data/ts-0/wal
uuid: "6d3c1300047541b7b7d267ad536912e1"
format_stamp: "Formatted at 2026-09-29 03:17:35 on dist-test-slave-c9qh"
server_key: "475e9b01895039c0e631c4a62ff0e0de"
server_key_iv: "16602f170eff1cc55a4ccbe52211285b"
server_key_version: "encryptionkey@0"
I20260929 03:17:35.787245 21825 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651852476208-21193-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651852476208-21193-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651852476208-21193-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:17:35.807862 21825 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:17:35.809837 21825 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:17:35.809954 21825 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:17:35.810148 21825 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:17:35.810588 21825 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:17:35.810627 21825 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:17:35.810681 21825 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:17:35.810711 21825 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:17:35.816418 21825 rpc_server.cc:307] RPC server started. Bound to: 127.20.178.65:33785
I20260929 03:17:35.816468 21953 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.178.65:33785 every 8 connection(s)
I20260929 03:17:35.816809 21825 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651852476208-21193-0/minicluster-data/ts-0/data/info.pb
I20260929 03:17:35.820379 21193 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuoaHbX/build/release/bin/kudu as pid 21825
I20260929 03:17:35.820477 21193 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651852476208-21193-0/minicluster-data/ts-0/wal/instance
I20260929 03:17:35.820683 21193 external_mini_cluster.cc:1428] Setting key 6d74b12ba37a13eacc1bee8c05dacaf4
Sep 29 03:17:35 dist-test-slave-c9qh krb5kdc[21718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651855, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.65@KRBTEST.COM for kudu/127.20.178.126@KRBTEST.COM
I20260929 03:17:35.827754 21808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:35.817548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.65:52331 (local address 127.20.178.126:35317)
0929 03:17:35.817632 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:35.817634 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:35.817818 (+ 184us) server_negotiation.cc:407] Connection header received
0929 03:17:35.818759 (+ 941us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:35.818760 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:35.818771 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:35.818829 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:17:35.819495 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:35.820029 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:35.821728 (+ 1699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:35.822070 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:35.822152 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:35.824794 (+ 2642us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:17:35.824796 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:17:35.824797 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:17:35.824804 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:17:35.826280 (+ 1476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:35.826718 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:35.826720 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:35.826720 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:35.826787 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:35.827056 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:35.827057 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:35.827058 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:35.827188 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:17:35.827263 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0929 03:17:35.827566 (+ 303us) server_negotiation.cc:299] Negotiation successful
0929 03:17:35.827660 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 03:17:35.827978 21956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:35.817504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:35317 (local address 127.20.178.65:52331)
0929 03:17:35.817698 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0929 03:17:35.817736 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 03:17:35.818631 (+ 895us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:17:35.818899 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:17:35.818903 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:17:35.819170 (+ 267us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:17:35.819366 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:35.819370 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:35.820146 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:35.820148 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:35.821537 (+ 1389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:35.821539 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:35.822163 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:35.822164 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:35.822287 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:35.822906 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:17:35.822917 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:17:35.824703 (+ 1786us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:17:35.826383 (+ 1680us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:17:35.826391 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:17:35.826394 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:17:35.826638 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:17:35.826860 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:17:35.826861 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:17:35.826862 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:17:35.826983 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:17:35.827266 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:17:35.827274 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:17:35.827415 (+ 141us) client_negotiation.cc:769] Sending connection context
0929 03:17:35.827540 (+ 125us) client_negotiation.cc:240] Negotiation successful
0929 03:17:35.827717 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":100,"threads_started":1}
I20260929 03:17:35.828624 21954 heartbeater.cc:347] Connected to a master server at 127.20.178.126:35317
I20260929 03:17:35.828758 21954 heartbeater.cc:464] Registering TS with master...
I20260929 03:17:35.828965 21954 heartbeater.cc:511] Master 127.20.178.126:35317 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.20.178.66@KRBTEST.COM; defaulting to no policy
I20260929 03:17:35.829562 21763 ts_manager.cc:194] Registered new tserver with Master: 6d3c1300047541b7b7d267ad536912e1 (127.20.178.65:33785)
I20260929 03:17:35.830168 21763 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.178.65@KRBTEST.COM'} at 127.20.178.65:52331
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.178.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.178.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.66.keytab.
Entry for principal kudu/127.20.178.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.66.keytab.
I20260929 03:17:35.847249 21193 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuoaHbX/build/release/bin/kudu
/tmp/dist-test-taskuoaHbX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651852476208-21193-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651852476208-21193-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651852476208-21193-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651852476208-21193-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.178.66:0
--local_ip_for_outbound_sockets=127.20.178.66
--webserver_interface=127.20.178.66
--webserver_port=0
--tserver_master_addrs=127.20.178.126:35317
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.66.keytab
--principal=kudu/127.20.178.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.178.84:46241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651852476208-21193-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651852476208-21193-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651852476208-21193-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:17:35.927804 21959 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:17:35.928012 21959 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:17:35.928036 21959 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:17:35.929280 21959 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:17:35.929324 21959 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:17:35.929373 21959 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.178.66
I20260929 03:17:35.930858 21959 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.178.84:46241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651852476208-21193-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651852476208-21193-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.66.keytab
--principal=kudu/127.20.178.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.178.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651852476208-21193-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.178.66
--webserver_port=0
--tserver_master_addrs=127.20.178.126:35317
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21959
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.178.66
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651852476208-21193-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:17:35.931103 21959 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:17:35.931367 21959 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:17:35.936291 21964 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:17:35.936306 21959 server_base.cc:1034] running on GCE node
W20260929 03:17:35.936357 21965 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:17:35.936520 21967 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:17:35.936785 21959 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:17:35.936985 21959 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:17:35.938158 21959 hybrid_clock.cc:648] HybridClock initialized: now 1790651855938138 us; error 40 us; skew 500 ppm
Sep 29 03:17:35 dist-test-slave-c9qh krb5kdc[21718](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651855, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:17:35.940639 21959 init.cc:377] Logged in from keytab as kudu/127.20.178.66@KRBTEST.COM (short username kudu)
I20260929 03:17:35.941103 21959 webserver.cc:460] Webserver started at http://127.20.178.66:37627/ using document root <none> and password file <none>
I20260929 03:17:35.941334 21959 fs_manager.cc:359] Metadata directory not provided
I20260929 03:17:35.941376 21959 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:17:35.941519 21959 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:17:35.942551 21959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651852476208-21193-0/minicluster-data/ts-1/data/instance:
uuid: "fc2561318ddc47f0b5591fbdfd906571"
format_stamp: "Formatted at 2026-09-29 03:17:35 on dist-test-slave-c9qh"
server_key: "d07f9e278059d30a3b184ff75515a514"
server_key_iv: "67e167fcf32b9d09c94055bd591fcd17"
server_key_version: "encryptionkey@0"
I20260929 03:17:35.942909 21959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651852476208-21193-0/minicluster-data/ts-1/wal/instance:
uuid: "fc2561318ddc47f0b5591fbdfd906571"
format_stamp: "Formatted at 2026-09-29 03:17:35 on dist-test-slave-c9qh"
server_key: "d07f9e278059d30a3b184ff75515a514"
server_key_iv: "67e167fcf32b9d09c94055bd591fcd17"
server_key_version: "encryptionkey@0"
I20260929 03:17:35.944191 21959 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 03:17:35.944922 21974 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:17:35.945094 21959 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 03:17:35.945174 21959 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651852476208-21193-0/minicluster-data/ts-1/data,/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651852476208-21193-0/minicluster-data/ts-1/wal
uuid: "fc2561318ddc47f0b5591fbdfd906571"
format_stamp: "Formatted at 2026-09-29 03:17:35 on dist-test-slave-c9qh"
server_key: "d07f9e278059d30a3b184ff75515a514"
server_key_iv: "67e167fcf32b9d09c94055bd591fcd17"
server_key_version: "encryptionkey@0"
I20260929 03:17:35.945264 21959 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651852476208-21193-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651852476208-21193-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651852476208-21193-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:17:35.969875 21959 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:17:35.971722 21959 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:17:35.971872 21959 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:17:35.972141 21959 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:17:35.972471 21959 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:17:35.972520 21959 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:17:35.972565 21959 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:17:35.972592 21959 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:17:35.978168 21959 rpc_server.cc:307] RPC server started. Bound to: 127.20.178.66:34119
I20260929 03:17:35.978214 22087 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.178.66:34119 every 8 connection(s)
I20260929 03:17:35.978516 21959 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651852476208-21193-0/minicluster-data/ts-1/data/info.pb
I20260929 03:17:35.982206 21193 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuoaHbX/build/release/bin/kudu as pid 21959
I20260929 03:17:35.982273 21193 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651852476208-21193-0/minicluster-data/ts-1/wal/instance
I20260929 03:17:35.982450 21193 external_mini_cluster.cc:1428] Setting key fa55b40daa73f920113265dd7f3f8f3e
Sep 29 03:17:35 dist-test-slave-c9qh krb5kdc[21718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651855, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.66@KRBTEST.COM for kudu/127.20.178.126@KRBTEST.COM
I20260929 03:17:35.988880 21808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:35.979195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.66:35831 (local address 127.20.178.126:35317)
0929 03:17:35.979259 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:35.979261 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:35.979470 (+ 209us) server_negotiation.cc:407] Connection header received
0929 03:17:35.980371 (+ 901us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:35.980372 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:35.980381 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:35.980440 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:17:35.981136 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:35.981729 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:35.982837 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:35.983121 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:35.983187 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:35.985814 (+ 2627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:17:35.985815 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:17:35.985815 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:17:35.985822 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:17:35.987261 (+ 1439us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:35.987780 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:35.987781 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:35.987781 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:35.987831 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:35.988120 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:35.988120 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:35.988121 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:35.988218 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:17:35.988268 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0929 03:17:35.988749 (+ 481us) server_negotiation.cc:299] Negotiation successful
0929 03:17:35.988817 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260929 03:17:35.989159 22090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:35.979173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:35317 (local address 127.20.178.66:35831)
0929 03:17:35.979344 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0929 03:17:35.979387 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 03:17:35.980282 (+ 895us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:17:35.980508 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:17:35.980512 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:17:35.980859 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:17:35.981053 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:35.981058 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:35.981809 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:35.981810 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:35.982768 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:35.982770 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:35.983219 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:35.983220 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:35.983377 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:35.983891 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:17:35.983901 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:17:35.985724 (+ 1823us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:17:35.987350 (+ 1626us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:17:35.987356 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:17:35.987359 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:17:35.987695 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:17:35.987894 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:17:35.987895 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:17:35.987896 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:17:35.988050 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:17:35.988285 (+ 235us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:17:35.988291 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:17:35.988472 (+ 181us) client_negotiation.cc:769] Sending connection context
0929 03:17:35.988707 (+ 235us) client_negotiation.cc:240] Negotiation successful
0929 03:17:35.988888 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":102,"threads_started":1}
I20260929 03:17:35.989710 22088 heartbeater.cc:347] Connected to a master server at 127.20.178.126:35317
I20260929 03:17:35.989818 22088 heartbeater.cc:464] Registering TS with master...
I20260929 03:17:35.990037 22088 heartbeater.cc:511] Master 127.20.178.126:35317 requested a full tablet report, sending...
I20260929 03:17:35.990442 21763 ts_manager.cc:194] Registered new tserver with Master: fc2561318ddc47f0b5591fbdfd906571 (127.20.178.66:34119)
I20260929 03:17:35.990845 21763 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.178.66@KRBTEST.COM'} at 127.20.178.66:35831
WARNING: no policy specified for kudu/127.20.178.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.178.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.178.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.67.keytab.
Entry for principal kudu/127.20.178.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.67.keytab.
I20260929 03:17:36.007886 21193 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuoaHbX/build/release/bin/kudu
/tmp/dist-test-taskuoaHbX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651852476208-21193-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651852476208-21193-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651852476208-21193-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651852476208-21193-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.178.67:0
--local_ip_for_outbound_sockets=127.20.178.67
--webserver_interface=127.20.178.67
--webserver_port=0
--tserver_master_addrs=127.20.178.126:35317
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.67.keytab
--principal=kudu/127.20.178.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.178.84:46241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651852476208-21193-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651852476208-21193-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651852476208-21193-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:17:36.087905 22093 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:17:36.088155 22093 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:17:36.088195 22093 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:17:36.089397 22093 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:17:36.089459 22093 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:17:36.089504 22093 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.178.67
I20260929 03:17:36.091035 22093 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.178.84:46241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651852476208-21193-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651852476208-21193-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.67.keytab
--principal=kudu/127.20.178.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.178.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651852476208-21193-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.178.67
--webserver_port=0
--tserver_master_addrs=127.20.178.126:35317
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22093
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.178.67
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651852476208-21193-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:17:36.091305 22093 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:17:36.091526 22093 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:17:36.096524 22098 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:17:36.096536 22101 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:17:36.096700 22099 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:17:36.096796 22093 server_base.cc:1034] running on GCE node
I20260929 03:17:36.097033 22093 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:17:36.097242 22093 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:17:36.098434 22093 hybrid_clock.cc:648] HybridClock initialized: now 1790651856098414 us; error 29 us; skew 500 ppm
Sep 29 03:17:36 dist-test-slave-c9qh krb5kdc[21718](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651856, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:17:36.100976 22093 init.cc:377] Logged in from keytab as kudu/127.20.178.67@KRBTEST.COM (short username kudu)
I20260929 03:17:36.101393 22093 webserver.cc:460] Webserver started at http://127.20.178.67:35305/ using document root <none> and password file <none>
I20260929 03:17:36.101616 22093 fs_manager.cc:359] Metadata directory not provided
I20260929 03:17:36.101691 22093 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:17:36.101845 22093 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:17:36.102900 22093 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651852476208-21193-0/minicluster-data/ts-2/data/instance:
uuid: "77365869e3dc48c09d344fd13ef7d5f0"
format_stamp: "Formatted at 2026-09-29 03:17:36 on dist-test-slave-c9qh"
server_key: "711fd4bb8100a3687b1e8bb9fa9729ee"
server_key_iv: "1adae3cc22674cf522c0ced5420e5c5a"
server_key_version: "encryptionkey@0"
I20260929 03:17:36.103260 22093 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651852476208-21193-0/minicluster-data/ts-2/wal/instance:
uuid: "77365869e3dc48c09d344fd13ef7d5f0"
format_stamp: "Formatted at 2026-09-29 03:17:36 on dist-test-slave-c9qh"
server_key: "711fd4bb8100a3687b1e8bb9fa9729ee"
server_key_iv: "1adae3cc22674cf522c0ced5420e5c5a"
server_key_version: "encryptionkey@0"
I20260929 03:17:36.104563 22093 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260929 03:17:36.105408 22108 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:17:36.105604 22093 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260929 03:17:36.105712 22093 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651852476208-21193-0/minicluster-data/ts-2/data,/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651852476208-21193-0/minicluster-data/ts-2/wal
uuid: "77365869e3dc48c09d344fd13ef7d5f0"
format_stamp: "Formatted at 2026-09-29 03:17:36 on dist-test-slave-c9qh"
server_key: "711fd4bb8100a3687b1e8bb9fa9729ee"
server_key_iv: "1adae3cc22674cf522c0ced5420e5c5a"
server_key_version: "encryptionkey@0"
I20260929 03:17:36.105795 22093 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651852476208-21193-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651852476208-21193-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651852476208-21193-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:17:36.130434 22093 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:17:36.132876 22093 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:17:36.133036 22093 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:17:36.133271 22093 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:17:36.133570 22093 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:17:36.133622 22093 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:17:36.133677 22093 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:17:36.133701 22093 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:17:36.139617 22093 rpc_server.cc:307] RPC server started. Bound to: 127.20.178.67:46645
I20260929 03:17:36.139657 22221 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.178.67:46645 every 8 connection(s)
I20260929 03:17:36.139981 22093 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651852476208-21193-0/minicluster-data/ts-2/data/info.pb
I20260929 03:17:36.142716 21193 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuoaHbX/build/release/bin/kudu as pid 22093
I20260929 03:17:36.142820 21193 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651852476208-21193-0/minicluster-data/ts-2/wal/instance
I20260929 03:17:36.142966 21193 external_mini_cluster.cc:1428] Setting key 5b35fe91ab2a89425134a193d0bd03c4
Sep 29 03:17:36 dist-test-slave-c9qh krb5kdc[21718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651856, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.67@KRBTEST.COM for kudu/127.20.178.126@KRBTEST.COM
I20260929 03:17:36.149358 21808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:36.140722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.67:40285 (local address 127.20.178.126:35317)
0929 03:17:36.140767 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:36.140775 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:36.141061 (+ 286us) server_negotiation.cc:407] Connection header received
0929 03:17:36.141978 (+ 917us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:36.141980 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:36.141990 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:36.142044 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:17:36.142672 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:36.143361 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:36.144460 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:36.144734 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:36.144823 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:36.146801 (+ 1978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:17:36.146802 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:17:36.146803 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:17:36.146809 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:17:36.148138 (+ 1329us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:36.148493 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:36.148494 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:36.148494 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:36.148547 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:36.148807 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:36.148808 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:36.148808 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:36.148925 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:17:36.148996 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0929 03:17:36.149231 (+ 235us) server_negotiation.cc:299] Negotiation successful
0929 03:17:36.149294 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260929 03:17:36.149526 22224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:36.140750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:35317 (local address 127.20.178.67:40285)
0929 03:17:36.140905 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0929 03:17:36.140974 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0929 03:17:36.141885 (+ 911us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:17:36.142115 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:17:36.142118 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:17:36.142378 (+ 260us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:17:36.142565 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:36.142568 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:36.143435 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:36.143436 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:36.144353 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:36.144355 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:36.144833 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:36.144834 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:36.144967 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:36.145405 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:17:36.145413 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:17:36.146723 (+ 1310us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:17:36.148204 (+ 1481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:17:36.148209 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:17:36.148211 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:17:36.148416 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:17:36.148603 (+ 187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:17:36.148604 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:17:36.148604 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:17:36.148751 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:17:36.148998 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:17:36.149002 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:17:36.149115 (+ 113us) client_negotiation.cc:769] Sending connection context
0929 03:17:36.149203 (+ 88us) client_negotiation.cc:240] Negotiation successful
0929 03:17:36.149320 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":68,"threads_started":1}
I20260929 03:17:36.150007 22222 heartbeater.cc:347] Connected to a master server at 127.20.178.126:35317
I20260929 03:17:36.150118 22222 heartbeater.cc:464] Registering TS with master...
I20260929 03:17:36.150300 22222 heartbeater.cc:511] Master 127.20.178.126:35317 requested a full tablet report, sending...
I20260929 03:17:36.150619 21763 ts_manager.cc:194] Registered new tserver with Master: 77365869e3dc48c09d344fd13ef7d5f0 (127.20.178.67:46645)
I20260929 03:17:36.150992 21763 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.178.67@KRBTEST.COM'} at 127.20.178.67:40285
I20260929 03:17:36.154563 21193 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 03:17:36.161892 21808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:36.155976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34708 (local address 127.20.178.126:35317)
0929 03:17:36.156037 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:36.156039 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:36.156167 (+ 128us) server_negotiation.cc:407] Connection header received
0929 03:17:36.156193 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:36.156193 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:36.156204 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:36.156273 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:17:36.157016 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:36.157480 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:36.158153 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:36.158330 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:36.158385 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:36.159591 (+ 1206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:17:36.159592 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:17:36.159592 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:17:36.159599 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:17:36.160869 (+ 1270us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:36.161187 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:36.161189 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:36.161189 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:36.161243 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:36.161405 (+ 162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:36.161405 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:36.161406 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:36.161539 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:17:36.161614 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0929 03:17:36.161714 (+ 100us) server_negotiation.cc:299] Negotiation successful
0929 03:17:36.161774 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260929 03:17:36.163930 21763 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34708:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 03:17:36.165494 21763 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 03:17:36.171871 22234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:36.167689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.66:34119 (local address 127.0.0.1:60014)
0929 03:17:36.167975 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0929 03:17:36.168018 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 03:17:36.168090 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:17:36.168313 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:17:36.168316 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:17:36.168318 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:17:36.168424 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:36.168427 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:36.169793 (+ 1366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:36.169794 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:36.170725 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:36.170728 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:36.171568 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:36.171569 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:36.171653 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:36.171670 (+ 17us) client_negotiation.cc:769] Sending connection context
0929 03:17:36.171736 (+ 66us) client_negotiation.cc:240] Negotiation successful
0929 03:17:36.171791 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"mutex_wait_us":39,"thread_start_us":75,"threads_started":1}
I20260929 03:17:36.171960 22236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:36.167807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60014 (local address 127.20.178.66:34119)
0929 03:17:36.168071 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:36.168075 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:36.168085 (+ 10us) server_negotiation.cc:407] Connection header received
0929 03:17:36.168202 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:36.168203 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:36.168219 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:36.168295 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:17:36.168518 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:36.169683 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:36.170811 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:36.171475 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:36.171551 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:36.171608 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0929 03:17:36.171832 (+ 224us) server_negotiation.cc:299] Negotiation successful
0929 03:17:36.171873 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":231,"thread_start_us":74,"threads_started":1}
I20260929 03:17:36.172387 22238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:36.168002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35556 (local address 127.20.178.65:33785)
0929 03:17:36.168197 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:36.168207 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:36.168472 (+ 265us) server_negotiation.cc:407] Connection header received
0929 03:17:36.168526 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:36.168528 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:36.168567 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:36.168671 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:17:36.168890 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:36.169564 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:36.171107 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:36.171646 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:36.171719 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:36.171776 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0929 03:17:36.172287 (+ 511us) server_negotiation.cc:299] Negotiation successful
0929 03:17:36.172330 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":84,"threads_started":1}
I20260929 03:17:36.172399 22235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:36.167797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59802 (local address 127.20.178.67:46645)
0929 03:17:36.167979 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:36.167985 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:36.168157 (+ 172us) server_negotiation.cc:407] Connection header received
0929 03:17:36.168199 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:36.168202 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:36.168219 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:36.168296 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:17:36.168518 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:36.169662 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:36.171063 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:36.171912 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:36.171972 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:36.172059 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0929 03:17:36.172282 (+ 223us) server_negotiation.cc:299] Negotiation successful
0929 03:17:36.172326 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":82,"threads_started":1}
I20260929 03:17:36.172626 22233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:36.167885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.65:33785 (local address 127.0.0.1:35556)
0929 03:17:36.168392 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0929 03:17:36.168401 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 03:17:36.168456 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:17:36.168760 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:17:36.168761 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:17:36.168763 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:17:36.168809 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:36.168810 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:36.169680 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:36.169681 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:36.171025 (+ 1344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:36.171027 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:36.171740 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:36.171741 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:36.171805 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:36.171810 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 03:17:36.171865 (+ 55us) client_negotiation.cc:240] Negotiation successful
0929 03:17:36.171909 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":57,"threads_started":1}
I20260929 03:17:36.172696 22022 tablet_service.cc:1467] Processing CreateTablet for tablet 7e9631f0dede48be9801c9569b20a886 (DEFAULT_TABLE table=test-table [id=1a6a4743ce3540258c82e682b1eb5738]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:17:36.173060 22237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:36.167689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.67:46645 (local address 127.0.0.1:59802)
0929 03:17:36.167992 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0929 03:17:36.168019 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0929 03:17:36.168090 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:17:36.168313 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:17:36.168316 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:17:36.168320 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:17:36.168424 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:36.168426 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:36.169769 (+ 1343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:36.169770 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:36.170958 (+ 1188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:36.170961 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:36.172091 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:36.172091 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:36.172154 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:36.172158 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 03:17:36.172210 (+ 52us) client_negotiation.cc:240] Negotiation successful
0929 03:17:36.172246 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"mutex_wait_us":94,"thread_start_us":60,"threads_started":1}
I20260929 03:17:36.173099 22022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e9631f0dede48be9801c9569b20a886. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:17:36.173454 21888 tablet_service.cc:1467] Processing CreateTablet for tablet 7e9631f0dede48be9801c9569b20a886 (DEFAULT_TABLE table=test-table [id=1a6a4743ce3540258c82e682b1eb5738]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:17:36.173767 21888 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e9631f0dede48be9801c9569b20a886. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:17:36.173938 22156 tablet_service.cc:1467] Processing CreateTablet for tablet 7e9631f0dede48be9801c9569b20a886 (DEFAULT_TABLE table=test-table [id=1a6a4743ce3540258c82e682b1eb5738]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:17:36.174230 22156 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e9631f0dede48be9801c9569b20a886. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:17:36.176461 22239 tablet_bootstrap.cc:492] T 7e9631f0dede48be9801c9569b20a886 P fc2561318ddc47f0b5591fbdfd906571: Bootstrap starting.
I20260929 03:17:36.176782 22240 tablet_bootstrap.cc:492] T 7e9631f0dede48be9801c9569b20a886 P 77365869e3dc48c09d344fd13ef7d5f0: Bootstrap starting.
I20260929 03:17:36.177078 22241 tablet_bootstrap.cc:492] T 7e9631f0dede48be9801c9569b20a886 P 6d3c1300047541b7b7d267ad536912e1: Bootstrap starting.
I20260929 03:17:36.177376 22239 tablet_bootstrap.cc:654] T 7e9631f0dede48be9801c9569b20a886 P fc2561318ddc47f0b5591fbdfd906571: Neither blocks nor log segments found. Creating new log.
I20260929 03:17:36.177440 22240 tablet_bootstrap.cc:654] T 7e9631f0dede48be9801c9569b20a886 P 77365869e3dc48c09d344fd13ef7d5f0: Neither blocks nor log segments found. Creating new log.
I20260929 03:17:36.177706 22241 tablet_bootstrap.cc:654] T 7e9631f0dede48be9801c9569b20a886 P 6d3c1300047541b7b7d267ad536912e1: Neither blocks nor log segments found. Creating new log.
I20260929 03:17:36.177927 22239 log.cc:824] T 7e9631f0dede48be9801c9569b20a886 P fc2561318ddc47f0b5591fbdfd906571: Log is configured to *not* fsync() on all Append() calls
I20260929 03:17:36.178056 22240 log.cc:824] T 7e9631f0dede48be9801c9569b20a886 P 77365869e3dc48c09d344fd13ef7d5f0: Log is configured to *not* fsync() on all Append() calls
I20260929 03:17:36.178280 22241 log.cc:824] T 7e9631f0dede48be9801c9569b20a886 P 6d3c1300047541b7b7d267ad536912e1: Log is configured to *not* fsync() on all Append() calls
I20260929 03:17:36.178817 22240 tablet_bootstrap.cc:492] T 7e9631f0dede48be9801c9569b20a886 P 77365869e3dc48c09d344fd13ef7d5f0: No bootstrap required, opened a new log
I20260929 03:17:36.178896 22240 ts_tablet_manager.cc:1397] T 7e9631f0dede48be9801c9569b20a886 P 77365869e3dc48c09d344fd13ef7d5f0: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 03:17:36.179001 22241 tablet_bootstrap.cc:492] T 7e9631f0dede48be9801c9569b20a886 P 6d3c1300047541b7b7d267ad536912e1: No bootstrap required, opened a new log
I20260929 03:17:36.179072 22241 ts_tablet_manager.cc:1397] T 7e9631f0dede48be9801c9569b20a886 P 6d3c1300047541b7b7d267ad536912e1: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 03:17:36.179085 22239 tablet_bootstrap.cc:492] T 7e9631f0dede48be9801c9569b20a886 P fc2561318ddc47f0b5591fbdfd906571: No bootstrap required, opened a new log
I20260929 03:17:36.179304 22239 ts_tablet_manager.cc:1397] T 7e9631f0dede48be9801c9569b20a886 P fc2561318ddc47f0b5591fbdfd906571: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260929 03:17:36.180359 22240 raft_consensus.cc:348] T 7e9631f0dede48be9801c9569b20a886 P 77365869e3dc48c09d344fd13ef7d5f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc2561318ddc47f0b5591fbdfd906571" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 34119 } } peers { permanent_uuid: "77365869e3dc48c09d344fd13ef7d5f0" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 46645 } } peers { permanent_uuid: "6d3c1300047541b7b7d267ad536912e1" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 33785 } }
I20260929 03:17:36.180493 22240 raft_consensus.cc:374] T 7e9631f0dede48be9801c9569b20a886 P 77365869e3dc48c09d344fd13ef7d5f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:17:36.180514 22240 raft_consensus.cc:729] T 7e9631f0dede48be9801c9569b20a886 P 77365869e3dc48c09d344fd13ef7d5f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77365869e3dc48c09d344fd13ef7d5f0, State: Initialized, Role: FOLLOWER
I20260929 03:17:36.180691 22240 consensus_queue.cc:260] T 7e9631f0dede48be9801c9569b20a886 P 77365869e3dc48c09d344fd13ef7d5f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc2561318ddc47f0b5591fbdfd906571" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 34119 } } peers { permanent_uuid: "77365869e3dc48c09d344fd13ef7d5f0" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 46645 } } peers { permanent_uuid: "6d3c1300047541b7b7d267ad536912e1" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 33785 } }
I20260929 03:17:36.181079 22240 ts_tablet_manager.cc:1428] T 7e9631f0dede48be9801c9569b20a886 P 77365869e3dc48c09d344fd13ef7d5f0: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260929 03:17:36.181043 22241 raft_consensus.cc:348] T 7e9631f0dede48be9801c9569b20a886 P 6d3c1300047541b7b7d267ad536912e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc2561318ddc47f0b5591fbdfd906571" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 34119 } } peers { permanent_uuid: "77365869e3dc48c09d344fd13ef7d5f0" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 46645 } } peers { permanent_uuid: "6d3c1300047541b7b7d267ad536912e1" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 33785 } }
I20260929 03:17:36.181214 22241 raft_consensus.cc:374] T 7e9631f0dede48be9801c9569b20a886 P 6d3c1300047541b7b7d267ad536912e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:17:36.181223 22222 heartbeater.cc:503] Master 127.20.178.126:35317 was elected leader, sending a full tablet report...
I20260929 03:17:36.181262 22241 raft_consensus.cc:729] T 7e9631f0dede48be9801c9569b20a886 P 6d3c1300047541b7b7d267ad536912e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d3c1300047541b7b7d267ad536912e1, State: Initialized, Role: FOLLOWER
I20260929 03:17:36.181370 22241 consensus_queue.cc:260] T 7e9631f0dede48be9801c9569b20a886 P 6d3c1300047541b7b7d267ad536912e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc2561318ddc47f0b5591fbdfd906571" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 34119 } } peers { permanent_uuid: "77365869e3dc48c09d344fd13ef7d5f0" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 46645 } } peers { permanent_uuid: "6d3c1300047541b7b7d267ad536912e1" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 33785 } }
I20260929 03:17:36.181635 22241 ts_tablet_manager.cc:1428] T 7e9631f0dede48be9801c9569b20a886 P 6d3c1300047541b7b7d267ad536912e1: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260929 03:17:36.181686 21954 heartbeater.cc:503] Master 127.20.178.126:35317 was elected leader, sending a full tablet report...
I20260929 03:17:36.181471 22239 raft_consensus.cc:348] T 7e9631f0dede48be9801c9569b20a886 P fc2561318ddc47f0b5591fbdfd906571 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc2561318ddc47f0b5591fbdfd906571" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 34119 } } peers { permanent_uuid: "77365869e3dc48c09d344fd13ef7d5f0" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 46645 } } peers { permanent_uuid: "6d3c1300047541b7b7d267ad536912e1" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 33785 } }
I20260929 03:17:36.181734 22239 raft_consensus.cc:374] T 7e9631f0dede48be9801c9569b20a886 P fc2561318ddc47f0b5591fbdfd906571 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:17:36.181768 22239 raft_consensus.cc:729] T 7e9631f0dede48be9801c9569b20a886 P fc2561318ddc47f0b5591fbdfd906571 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc2561318ddc47f0b5591fbdfd906571, State: Initialized, Role: FOLLOWER
I20260929 03:17:36.181890 22239 consensus_queue.cc:260] T 7e9631f0dede48be9801c9569b20a886 P fc2561318ddc47f0b5591fbdfd906571 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc2561318ddc47f0b5591fbdfd906571" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 34119 } } peers { permanent_uuid: "77365869e3dc48c09d344fd13ef7d5f0" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 46645 } } peers { permanent_uuid: "6d3c1300047541b7b7d267ad536912e1" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 33785 } }
I20260929 03:17:36.182155 22088 heartbeater.cc:503] Master 127.20.178.126:35317 was elected leader, sending a full tablet report...
I20260929 03:17:36.182160 22239 ts_tablet_manager.cc:1428] T 7e9631f0dede48be9801c9569b20a886 P fc2561318ddc47f0b5591fbdfd906571: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260929 03:17:36.225224 22247 raft_consensus.cc:482] T 7e9631f0dede48be9801c9569b20a886 P fc2561318ddc47f0b5591fbdfd906571 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:17:36.225340 22247 raft_consensus.cc:504] T 7e9631f0dede48be9801c9569b20a886 P fc2561318ddc47f0b5591fbdfd906571 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc2561318ddc47f0b5591fbdfd906571" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 34119 } } peers { permanent_uuid: "77365869e3dc48c09d344fd13ef7d5f0" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 46645 } } peers { permanent_uuid: "6d3c1300047541b7b7d267ad536912e1" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 33785 } }
I20260929 03:17:36.225792 22247 leader_election.cc:288] T 7e9631f0dede48be9801c9569b20a886 P fc2561318ddc47f0b5591fbdfd906571 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77365869e3dc48c09d344fd13ef7d5f0 (127.20.178.67:46645), 6d3c1300047541b7b7d267ad536912e1 (127.20.178.65:33785)
I20260929 03:17:36.228513 22235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:36.225992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.66:37235 (local address 127.20.178.67:46645)
0929 03:17:36.226040 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:36.226042 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:36.226054 (+ 12us) server_negotiation.cc:407] Connection header received
0929 03:17:36.226125 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:36.226126 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:36.226143 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:36.226203 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:17:36.226387 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:36.226861 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:36.227798 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:36.228170 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:36.228259 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:36.228268 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0929 03:17:36.228415 (+ 147us) server_negotiation.cc:299] Negotiation successful
0929 03:17:36.228449 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260929 03:17:36.228514 22090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:36.225935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.67:46645 (local address 127.20.178.66:37235)
0929 03:17:36.226004 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0929 03:17:36.226017 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 03:17:36.226064 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:17:36.226233 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:17:36.226234 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:17:36.226236 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:17:36.226304 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:36.226311 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:36.226949 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:36.226949 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:36.227703 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:36.227704 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:36.228255 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:36.228256 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:36.228331 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:36.228334 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 03:17:36.228409 (+ 75us) client_negotiation.cc:240] Negotiation successful
0929 03:17:36.228447 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260929 03:17:36.228715 22238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:36.226116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.66:48793 (local address 127.20.178.65:33785)
0929 03:17:36.226161 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:36.226164 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:36.226262 (+ 98us) server_negotiation.cc:407] Connection header received
0929 03:17:36.226327 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:36.226333 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:36.226342 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:36.226412 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:17:36.226563 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:36.227021 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:36.227956 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:36.228321 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:36.228402 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:36.228410 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0929 03:17:36.228556 (+ 146us) server_negotiation.cc:299] Negotiation successful
0929 03:17:36.228593 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260929 03:17:36.228884 22248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:36.225957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.65:33785 (local address 127.20.178.66:48793)
0929 03:17:36.226189 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0929 03:17:36.226197 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0929 03:17:36.226265 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:17:36.226415 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:17:36.226416 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:17:36.226417 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:17:36.226495 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:36.226496 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:36.227106 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:36.227107 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:36.227868 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:36.227870 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:36.228401 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:36.228402 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:36.228479 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:36.228482 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 03:17:36.228540 (+ 58us) client_negotiation.cc:240] Negotiation successful
0929 03:17:36.228728 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":130,"threads_started":1}
I20260929 03:17:36.228955 22176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e9631f0dede48be9801c9569b20a886" candidate_uuid: "fc2561318ddc47f0b5591fbdfd906571" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77365869e3dc48c09d344fd13ef7d5f0" is_pre_election: true
W20260929 03:17:36.229032 22089 tablet.cc:2369] T 7e9631f0dede48be9801c9569b20a886 P fc2561318ddc47f0b5591fbdfd906571: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 03:17:36.229121 22176 raft_consensus.cc:2454] T 7e9631f0dede48be9801c9569b20a886 P 77365869e3dc48c09d344fd13ef7d5f0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc2561318ddc47f0b5591fbdfd906571 in term 0.
I20260929 03:17:36.229396 21977 leader_election.cc:302] T 7e9631f0dede48be9801c9569b20a886 P fc2561318ddc47f0b5591fbdfd906571 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77365869e3dc48c09d344fd13ef7d5f0, fc2561318ddc47f0b5591fbdfd906571; no voters:
I20260929 03:17:36.229377 21908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e9631f0dede48be9801c9569b20a886" candidate_uuid: "fc2561318ddc47f0b5591fbdfd906571" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d3c1300047541b7b7d267ad536912e1" is_pre_election: true
I20260929 03:17:36.229514 21908 raft_consensus.cc:2454] T 7e9631f0dede48be9801c9569b20a886 P 6d3c1300047541b7b7d267ad536912e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc2561318ddc47f0b5591fbdfd906571 in term 0.
I20260929 03:17:36.229533 22247 raft_consensus.cc:2781] T 7e9631f0dede48be9801c9569b20a886 P fc2561318ddc47f0b5591fbdfd906571 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 03:17:36.229601 22247 raft_consensus.cc:482] T 7e9631f0dede48be9801c9569b20a886 P fc2561318ddc47f0b5591fbdfd906571 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 03:17:36.229626 22247 raft_consensus.cc:3037] T 7e9631f0dede48be9801c9569b20a886 P fc2561318ddc47f0b5591fbdfd906571 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:17:36.230324 22247 raft_consensus.cc:504] T 7e9631f0dede48be9801c9569b20a886 P fc2561318ddc47f0b5591fbdfd906571 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc2561318ddc47f0b5591fbdfd906571" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 34119 } } peers { permanent_uuid: "77365869e3dc48c09d344fd13ef7d5f0" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 46645 } } peers { permanent_uuid: "6d3c1300047541b7b7d267ad536912e1" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 33785 } }
I20260929 03:17:36.230463 22247 leader_election.cc:288] T 7e9631f0dede48be9801c9569b20a886 P fc2561318ddc47f0b5591fbdfd906571 [CANDIDATE]: Term 1 election: Requested vote from peers 77365869e3dc48c09d344fd13ef7d5f0 (127.20.178.67:46645), 6d3c1300047541b7b7d267ad536912e1 (127.20.178.65:33785)
I20260929 03:17:36.230589 22176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e9631f0dede48be9801c9569b20a886" candidate_uuid: "fc2561318ddc47f0b5591fbdfd906571" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77365869e3dc48c09d344fd13ef7d5f0"
I20260929 03:17:36.230649 21908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e9631f0dede48be9801c9569b20a886" candidate_uuid: "fc2561318ddc47f0b5591fbdfd906571" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d3c1300047541b7b7d267ad536912e1"
I20260929 03:17:36.230679 22176 raft_consensus.cc:3037] T 7e9631f0dede48be9801c9569b20a886 P 77365869e3dc48c09d344fd13ef7d5f0 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:17:36.230711 21908 raft_consensus.cc:3037] T 7e9631f0dede48be9801c9569b20a886 P 6d3c1300047541b7b7d267ad536912e1 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:17:36.231539 21908 raft_consensus.cc:2454] T 7e9631f0dede48be9801c9569b20a886 P 6d3c1300047541b7b7d267ad536912e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc2561318ddc47f0b5591fbdfd906571 in term 1.
I20260929 03:17:36.231539 22176 raft_consensus.cc:2454] T 7e9631f0dede48be9801c9569b20a886 P 77365869e3dc48c09d344fd13ef7d5f0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc2561318ddc47f0b5591fbdfd906571 in term 1.
I20260929 03:17:36.231732 21978 leader_election.cc:302] T 7e9631f0dede48be9801c9569b20a886 P fc2561318ddc47f0b5591fbdfd906571 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d3c1300047541b7b7d267ad536912e1, fc2561318ddc47f0b5591fbdfd906571; no voters:
I20260929 03:17:36.231838 22247 raft_consensus.cc:2781] T 7e9631f0dede48be9801c9569b20a886 P fc2561318ddc47f0b5591fbdfd906571 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:17:36.231993 22247 raft_consensus.cc:686] T 7e9631f0dede48be9801c9569b20a886 P fc2561318ddc47f0b5591fbdfd906571 [term 1 LEADER]: Becoming Leader. State: Replica: fc2561318ddc47f0b5591fbdfd906571, State: Running, Role: LEADER
I20260929 03:17:36.232096 22247 consensus_queue.cc:237] T 7e9631f0dede48be9801c9569b20a886 P fc2561318ddc47f0b5591fbdfd906571 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc2561318ddc47f0b5591fbdfd906571" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 34119 } } peers { permanent_uuid: "77365869e3dc48c09d344fd13ef7d5f0" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 46645 } } peers { permanent_uuid: "6d3c1300047541b7b7d267ad536912e1" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 33785 } }
I20260929 03:17:36.233004 21762 catalog_manager.cc:5495] T 7e9631f0dede48be9801c9569b20a886 P fc2561318ddc47f0b5591fbdfd906571 reported cstate change: term changed from 0 to 1, leader changed from <none> to fc2561318ddc47f0b5591fbdfd906571 (127.20.178.66). New cstate: current_term: 1 leader_uuid: "fc2561318ddc47f0b5591fbdfd906571" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc2561318ddc47f0b5591fbdfd906571" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 34119 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "77365869e3dc48c09d344fd13ef7d5f0" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 46645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6d3c1300047541b7b7d267ad536912e1" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 33785 } health_report { overall_health: UNKNOWN } } }
Sep 29 03:17:36 dist-test-slave-c9qh krb5kdc[21718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651855, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.178.65@KRBTEST.COM
I20260929 03:17:36.248152 22238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:36.240727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35562 (local address 127.20.178.65:33785)
0929 03:17:36.240777 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:36.240779 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:36.240925 (+ 146us) server_negotiation.cc:407] Connection header received
0929 03:17:36.240962 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:36.240962 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:36.240971 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:36.241020 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:17:36.241612 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:36.242060 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:36.242734 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:36.242917 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:36.242965 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:36.245419 (+ 2454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:17:36.245428 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:17:36.245428 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:17:36.245435 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:17:36.247076 (+ 1641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:36.247421 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:36.247429 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:36.247430 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:36.247484 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:36.247644 (+ 160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:36.247644 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:36.247669 (+ 25us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:36.247792 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:17:36.247923 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0929 03:17:36.247961 (+ 38us) server_negotiation.cc:299] Negotiation successful
0929 03:17:36.248020 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
Sep 29 03:17:36 dist-test-slave-c9qh krb5kdc[21718](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651856, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 03:17:36 dist-test-slave-c9qh krb5kdc[21718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651856, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.20.178.65@KRBTEST.COM
I20260929 03:17:36.272004 22238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:36.263945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35566 (local address 127.20.178.65:33785)
0929 03:17:36.264030 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:36.264033 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:36.264120 (+ 87us) server_negotiation.cc:407] Connection header received
0929 03:17:36.264220 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:36.264221 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:36.264231 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:36.264291 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:17:36.264905 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:36.265407 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:36.266149 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:36.266349 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:36.266434 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:36.268589 (+ 2155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:17:36.268590 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:17:36.268591 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:17:36.268597 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:17:36.270543 (+ 1946us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:36.270930 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:36.270931 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:36.270931 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:36.271013 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:36.271267 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:36.271268 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:36.271269 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:36.271393 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:17:36.271500 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0929 03:17:36.271801 (+ 301us) server_negotiation.cc:299] Negotiation successful
0929 03:17:36.271899 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
Sep 29 03:17:36 dist-test-slave-c9qh krb5kdc[21718](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651856, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 03:17:36 dist-test-slave-c9qh krb5kdc[21718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651856, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.178.65@KRBTEST.COM
I20260929 03:17:36.302501 22238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:36.293363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35572 (local address 127.20.178.65:33785)
0929 03:17:36.293432 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:36.293434 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:36.293563 (+ 129us) server_negotiation.cc:407] Connection header received
0929 03:17:36.293663 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:36.293664 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:36.293673 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:36.293747 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:17:36.294513 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:36.295324 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:36.296055 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:36.296255 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:36.296345 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:36.299502 (+ 3157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:17:36.299504 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:17:36.299504 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:17:36.299513 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:17:36.301206 (+ 1693us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:36.301569 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:36.301570 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:36.301571 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:36.301631 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:36.301883 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:36.301884 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:36.301884 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:36.302014 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:17:36.302126 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0929 03:17:36.302308 (+ 182us) server_negotiation.cc:299] Negotiation successful
0929 03:17:36.302389 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
W20260929 03:17:36.303001 21868 tablet_service.cc:2908] Rejecting scan request for tablet 7e9631f0dede48be9801c9569b20a886: Uninitialized: clean time has not yet been initialized
I20260929 03:17:36.304037 21193 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuoaHbX/build/release/bin/kudu with pid 21825
I20260929 03:17:36.311347 21193 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuoaHbX/build/release/bin/kudu with pid 21959
I20260929 03:17:36.317775 21193 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuoaHbX/build/release/bin/kudu with pid 22093
I20260929 03:17:36.323489 21193 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuoaHbX/build/release/bin/kudu with pid 21731
2026-09-29T03:17:36Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (1994 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651852476208-21193-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:17:36 dist-test-slave-c9qh krb5kdc[22263](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:17:36 dist-test-slave-c9qh krb5kdc[22263](info): set up 2 sockets
Sep 29 03:17:36 dist-test-slave-c9qh krb5kdc[22263](info): commencing operation
krb5kdc: starting...
W20260929 03:17:37.352200 21193 mini_kdc.cc:121] Time spent starting KDC: real 1.007s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651852476208-21193-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651852476208-21193-0/krb5kdc/test-user.keytab.
Sep 29 03:17:37 dist-test-slave-c9qh krb5kdc[22263](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651857, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:17:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:17:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.178.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.178.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.178.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.126.keytab.
Entry for principal kudu/127.20.178.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.126.keytab.
I20260929 03:17:37.467227 21193 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuoaHbX/build/release/bin/kudu
/tmp/dist-test-taskuoaHbX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651852476208-21193-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651852476208-21193-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651852476208-21193-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651852476208-21193-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.178.126:36403
--webserver_interface=127.20.178.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.126.keytab
--principal=kudu/127.20.178.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.178.84:38049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.178.126:36403
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651852476208-21193-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651852476208-21193-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651852476208-21193-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:17:37.546860 22276 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:17:37.547060 22276 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:17:37.547082 22276 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:17:37.548224 22276 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:17:37.548264 22276 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260929 03:17:37.548278 22276 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:17:37.548316 22276 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:17:37.548344 22276 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:17:37.548360 22276 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:17:37.549870 22276 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.178.84:38049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651852476208-21193-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651852476208-21193-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.178.126:36403
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.126.keytab
--principal=kudu/127.20.178.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.178.126:36403
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651852476208-21193-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.178.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22276
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651852476208-21193-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:17:37.550125 22276 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:17:37.550359 22276 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:17:37.555294 22284 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:17:37.555251 22282 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:17:37.555429 22281 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:17:37.555541 22276 server_base.cc:1034] running on GCE node
I20260929 03:17:37.555768 22276 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:17:37.555995 22276 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:17:37.557132 22276 hybrid_clock.cc:648] HybridClock initialized: now 1790651857557121 us; error 28 us; skew 500 ppm
Sep 29 03:17:37 dist-test-slave-c9qh krb5kdc[22263](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651857, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:17:37.565656 22276 init.cc:377] Logged in from keytab as kudu/127.20.178.126@KRBTEST.COM (short username kudu)
I20260929 03:17:37.566143 22276 webserver.cc:460] Webserver started at http://127.20.178.126:40435/ using document root <none> and password file <none>
I20260929 03:17:37.566380 22276 fs_manager.cc:359] Metadata directory not provided
I20260929 03:17:37.566444 22276 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:17:37.566617 22276 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:17:37.567673 22276 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651852476208-21193-0/minicluster-data/master-0/data/instance:
uuid: "721722c2f2ff42e788510074c1a2d3fd"
format_stamp: "Formatted at 2026-09-29 03:17:37 on dist-test-slave-c9qh"
server_key: "984ff7e206ccd61afd6602c38332f91d"
server_key_iv: "016cf9586b1453c224caebfff5088dee"
server_key_version: "encryptionkey@0"
I20260929 03:17:37.568028 22276 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651852476208-21193-0/minicluster-data/master-0/wal/instance:
uuid: "721722c2f2ff42e788510074c1a2d3fd"
format_stamp: "Formatted at 2026-09-29 03:17:37 on dist-test-slave-c9qh"
server_key: "984ff7e206ccd61afd6602c38332f91d"
server_key_iv: "016cf9586b1453c224caebfff5088dee"
server_key_version: "encryptionkey@0"
I20260929 03:17:37.569416 22276 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260929 03:17:37.570143 22291 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:17:37.570313 22276 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 03:17:37.570416 22276 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651852476208-21193-0/minicluster-data/master-0/data,/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651852476208-21193-0/minicluster-data/master-0/wal
uuid: "721722c2f2ff42e788510074c1a2d3fd"
format_stamp: "Formatted at 2026-09-29 03:17:37 on dist-test-slave-c9qh"
server_key: "984ff7e206ccd61afd6602c38332f91d"
server_key_iv: "016cf9586b1453c224caebfff5088dee"
server_key_version: "encryptionkey@0"
I20260929 03:17:37.570487 22276 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651852476208-21193-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651852476208-21193-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651852476208-21193-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:17:37.580374 22276 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:17:37.582664 22276 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:17:37.582823 22276 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:17:37.587469 22276 rpc_server.cc:307] RPC server started. Bound to: 127.20.178.126:36403
I20260929 03:17:37.587525 22353 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.178.126:36403 every 8 connection(s)
I20260929 03:17:37.587841 22276 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651852476208-21193-0/minicluster-data/master-0/data/info.pb
I20260929 03:17:37.588413 22354 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:17:37.590668 22354 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 721722c2f2ff42e788510074c1a2d3fd: Bootstrap starting.
I20260929 03:17:37.591260 22354 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 721722c2f2ff42e788510074c1a2d3fd: Neither blocks nor log segments found. Creating new log.
I20260929 03:17:37.591548 22354 log.cc:824] T 00000000000000000000000000000000 P 721722c2f2ff42e788510074c1a2d3fd: Log is configured to *not* fsync() on all Append() calls
I20260929 03:17:37.591925 21193 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuoaHbX/build/release/bin/kudu as pid 22276
I20260929 03:17:37.592022 21193 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651852476208-21193-0/minicluster-data/master-0/wal/instance
I20260929 03:17:37.592180 21193 external_mini_cluster.cc:1428] Setting key b265ddc82ce6fc30d74c28e9a918d337
I20260929 03:17:37.592353 22354 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 721722c2f2ff42e788510074c1a2d3fd: No bootstrap required, opened a new log
I20260929 03:17:37.595129 22354 raft_consensus.cc:348] T 00000000000000000000000000000000 P 721722c2f2ff42e788510074c1a2d3fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "721722c2f2ff42e788510074c1a2d3fd" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 36403 } }
I20260929 03:17:37.595265 22354 raft_consensus.cc:374] T 00000000000000000000000000000000 P 721722c2f2ff42e788510074c1a2d3fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:17:37.595288 22354 raft_consensus.cc:729] T 00000000000000000000000000000000 P 721722c2f2ff42e788510074c1a2d3fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 721722c2f2ff42e788510074c1a2d3fd, State: Initialized, Role: FOLLOWER
I20260929 03:17:37.595418 22354 consensus_queue.cc:260] T 00000000000000000000000000000000 P 721722c2f2ff42e788510074c1a2d3fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "721722c2f2ff42e788510074c1a2d3fd" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 36403 } }
I20260929 03:17:37.595490 22354 raft_consensus.cc:388] T 00000000000000000000000000000000 P 721722c2f2ff42e788510074c1a2d3fd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:17:37.595513 22354 raft_consensus.cc:482] T 00000000000000000000000000000000 P 721722c2f2ff42e788510074c1a2d3fd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:17:37.595579 22354 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 721722c2f2ff42e788510074c1a2d3fd [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:17:37.596336 22354 raft_consensus.cc:504] T 00000000000000000000000000000000 P 721722c2f2ff42e788510074c1a2d3fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "721722c2f2ff42e788510074c1a2d3fd" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 36403 } }
I20260929 03:17:37.596446 22354 leader_election.cc:302] T 00000000000000000000000000000000 P 721722c2f2ff42e788510074c1a2d3fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 721722c2f2ff42e788510074c1a2d3fd; no voters:
I20260929 03:17:37.596597 22354 leader_election.cc:288] T 00000000000000000000000000000000 P 721722c2f2ff42e788510074c1a2d3fd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:17:37.596700 22359 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 721722c2f2ff42e788510074c1a2d3fd [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:17:37.596863 22354 sys_catalog.cc:564] T 00000000000000000000000000000000 P 721722c2f2ff42e788510074c1a2d3fd [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:17:37.596865 22359 raft_consensus.cc:686] T 00000000000000000000000000000000 P 721722c2f2ff42e788510074c1a2d3fd [term 1 LEADER]: Becoming Leader. State: Replica: 721722c2f2ff42e788510074c1a2d3fd, State: Running, Role: LEADER
I20260929 03:17:37.596987 22359 consensus_queue.cc:237] T 00000000000000000000000000000000 P 721722c2f2ff42e788510074c1a2d3fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "721722c2f2ff42e788510074c1a2d3fd" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 36403 } }
I20260929 03:17:37.597291 22360 sys_catalog.cc:455] T 00000000000000000000000000000000 P 721722c2f2ff42e788510074c1a2d3fd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "721722c2f2ff42e788510074c1a2d3fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "721722c2f2ff42e788510074c1a2d3fd" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 36403 } } }
I20260929 03:17:37.597290 22361 sys_catalog.cc:455] T 00000000000000000000000000000000 P 721722c2f2ff42e788510074c1a2d3fd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 721722c2f2ff42e788510074c1a2d3fd. Latest consensus state: current_term: 1 leader_uuid: "721722c2f2ff42e788510074c1a2d3fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "721722c2f2ff42e788510074c1a2d3fd" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 36403 } } }
I20260929 03:17:37.597507 22361 sys_catalog.cc:458] T 00000000000000000000000000000000 P 721722c2f2ff42e788510074c1a2d3fd [sys.catalog]: This master's current role is: LEADER
I20260929 03:17:37.597718 22360 sys_catalog.cc:458] T 00000000000000000000000000000000 P 721722c2f2ff42e788510074c1a2d3fd [sys.catalog]: This master's current role is: LEADER
I20260929 03:17:37.597874 22368 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:17:37.598374 22368 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 29 03:17:37 dist-test-slave-c9qh krb5kdc[22263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651857, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.178.126@KRBTEST.COM
I20260929 03:17:37.600757 22368 catalog_manager.cc:1347] Generated new cluster ID: 372cbe3d4d674a18a245f1e4045799b9
I20260929 03:17:37.600808 22368 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:17:37.604750 22358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:37.592756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37928 (local address 127.20.178.126:36403)
0929 03:17:37.592911 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:37.592919 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:37.592955 (+ 36us) server_negotiation.cc:407] Connection header received
0929 03:17:37.593681 (+ 726us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:37.593688 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:37.593743 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:37.593857 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:17:37.594699 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:37.595774 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:37.596900 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:37.597647 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:37.597743 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:37.600598 (+ 2855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:17:37.600602 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:17:37.600603 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:17:37.600615 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:17:37.602629 (+ 2014us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:37.603047 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:37.603050 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:37.603050 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:37.603158 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:37.603363 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:37.603364 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:37.603364 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:37.603668 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:17:37.604281 (+ 613us) server_negotiation.cc:1035] Waiting for connection context
0929 03:17:37.604357 (+ 76us) server_negotiation.cc:299] Negotiation successful
0929 03:17:37.604497 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":86,"threads_started":1}
I20260929 03:17:37.623411 22368 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:17:37.623980 22368 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:17:37.630772 22368 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 721722c2f2ff42e788510074c1a2d3fd: Generated new TSK 0
I20260929 03:17:37.630992 22368 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.178.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.178.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.178.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.65.keytab.
Entry for principal kudu/127.20.178.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.65.keytab.
I20260929 03:17:37.657626 21193 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuoaHbX/build/release/bin/kudu
/tmp/dist-test-taskuoaHbX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651852476208-21193-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651852476208-21193-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651852476208-21193-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651852476208-21193-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.178.65:0
--local_ip_for_outbound_sockets=127.20.178.65
--webserver_interface=127.20.178.65
--webserver_port=0
--tserver_master_addrs=127.20.178.126:36403
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.65.keytab
--principal=kudu/127.20.178.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.178.84:38049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651852476208-21193-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651852476208-21193-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651852476208-21193-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:17:37.738503 22380 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:17:37.738706 22380 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:17:37.738730 22380 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:17:37.739801 22380 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:17:37.739833 22380 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:17:37.739854 22380 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 03:17:37.739871 22380 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.178.65
I20260929 03:17:37.741250 22380 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.178.84:38049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651852476208-21193-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651852476208-21193-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.65.keytab
--principal=kudu/127.20.178.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.178.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651852476208-21193-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.178.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.20.178.126:36403
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22380
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.178.65
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651852476208-21193-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:17:37.741421 22380 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:17:37.741652 22380 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 03:17:37.746709 22380 server_base.cc:1034] running on GCE node
W20260929 03:17:37.746650 22386 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:17:37.746637 22385 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:17:37.746687 22388 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:17:37.747149 22380 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:17:37.747432 22380 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:17:37.748582 22380 hybrid_clock.cc:648] HybridClock initialized: now 1790651857748573 us; error 29 us; skew 500 ppm
Sep 29 03:17:37 dist-test-slave-c9qh krb5kdc[22263](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651857, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:17:37.751102 22380 init.cc:377] Logged in from keytab as kudu/127.20.178.65@KRBTEST.COM (short username kudu)
I20260929 03:17:37.751576 22380 webserver.cc:460] Webserver started at http://127.20.178.65:41521/ using document root <none> and password file <none>
I20260929 03:17:37.751847 22380 fs_manager.cc:359] Metadata directory not provided
I20260929 03:17:37.751917 22380 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:17:37.752067 22380 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:17:37.753180 22380 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651852476208-21193-0/minicluster-data/ts-0/data/instance:
uuid: "64ec142b9e1f41cc808a076c2fcf4409"
format_stamp: "Formatted at 2026-09-29 03:17:37 on dist-test-slave-c9qh"
server_key: "86c91a927e6732260564ad94341ad712"
server_key_iv: "c72e4f8dfe2a2bf94ae831556f53d316"
server_key_version: "encryptionkey@0"
I20260929 03:17:37.753531 22380 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651852476208-21193-0/minicluster-data/ts-0/wal/instance:
uuid: "64ec142b9e1f41cc808a076c2fcf4409"
format_stamp: "Formatted at 2026-09-29 03:17:37 on dist-test-slave-c9qh"
server_key: "86c91a927e6732260564ad94341ad712"
server_key_iv: "c72e4f8dfe2a2bf94ae831556f53d316"
server_key_version: "encryptionkey@0"
I20260929 03:17:37.754813 22380 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 03:17:37.755486 22395 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:17:37.755671 22380 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 03:17:37.755770 22380 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651852476208-21193-0/minicluster-data/ts-0/data,/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651852476208-21193-0/minicluster-data/ts-0/wal
uuid: "64ec142b9e1f41cc808a076c2fcf4409"
format_stamp: "Formatted at 2026-09-29 03:17:37 on dist-test-slave-c9qh"
server_key: "86c91a927e6732260564ad94341ad712"
server_key_iv: "c72e4f8dfe2a2bf94ae831556f53d316"
server_key_version: "encryptionkey@0"
I20260929 03:17:37.755846 22380 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651852476208-21193-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651852476208-21193-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651852476208-21193-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:17:37.768739 22380 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:17:37.770962 22380 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:17:37.771119 22380 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:17:37.771695 22380 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:17:37.771746 22380 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:17:37.771795 22380 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:17:37.771822 22380 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 03:17:37 dist-test-slave-c9qh krb5kdc[22263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651857, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.65@KRBTEST.COM for kudu/127.20.178.126@KRBTEST.COM
I20260929 03:17:37.779072 22380 rpc_server.cc:307] RPC server started. Bound to: 127.20.178.65:32925
I20260929 03:17:37.779111 22510 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.178.65:32925 every 8 connection(s)
I20260929 03:17:37.779421 22380 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651852476208-21193-0/minicluster-data/ts-0/data/info.pb
I20260929 03:17:37.781628 22358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:37.772080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.65:51197 (local address 127.20.178.126:36403)
0929 03:17:37.772155 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:37.772157 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:37.772381 (+ 224us) server_negotiation.cc:407] Connection header received
0929 03:17:37.772930 (+ 549us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:37.772931 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:37.772944 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:37.773025 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:17:37.773647 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:37.774557 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:37.775615 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:37.775912 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:37.776012 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:37.778571 (+ 2559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:17:37.778572 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:17:37.778573 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:17:37.778583 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:17:37.780177 (+ 1594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:37.780626 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:37.780627 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:37.780628 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:37.780705 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:37.780965 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:37.780966 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:37.780967 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:37.781100 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:17:37.781173 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 03:17:37.781448 (+ 275us) server_negotiation.cc:299] Negotiation successful
0929 03:17:37.781535 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 03:17:37.781826 22404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:37.772085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:36403 (local address 127.20.178.65:51197)
0929 03:17:37.772250 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0929 03:17:37.772291 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 03:17:37.772833 (+ 542us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:17:37.773088 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:17:37.773091 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:17:37.773338 (+ 247us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:17:37.773572 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:37.773577 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:37.774650 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:37.774651 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:37.775483 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:37.775486 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:37.776021 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:37.776023 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:37.776201 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:37.776775 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:17:37.776787 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:17:37.778439 (+ 1652us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:17:37.780264 (+ 1825us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:17:37.780270 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:17:37.780272 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:17:37.780542 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:17:37.780781 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:17:37.780782 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:17:37.780782 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:17:37.780895 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:17:37.781175 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:17:37.781183 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:17:37.781319 (+ 136us) client_negotiation.cc:769] Sending connection context
0929 03:17:37.781426 (+ 107us) client_negotiation.cc:240] Negotiation successful
0929 03:17:37.781594 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":96,"threads_started":1}
I20260929 03:17:37.782210 21193 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuoaHbX/build/release/bin/kudu as pid 22380
I20260929 03:17:37.782310 21193 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651852476208-21193-0/minicluster-data/ts-0/wal/instance
I20260929 03:17:37.782310 22511 heartbeater.cc:347] Connected to a master server at 127.20.178.126:36403
I20260929 03:17:37.782387 22511 heartbeater.cc:464] Registering TS with master...
I20260929 03:17:37.782528 21193 external_mini_cluster.cc:1428] Setting key ace330b8544d180c2f4e87be1e30fd38
I20260929 03:17:37.782582 22511 heartbeater.cc:511] Master 127.20.178.126:36403 requested a full tablet report, sending...
I20260929 03:17:37.783095 22308 ts_manager.cc:194] Registered new tserver with Master: 64ec142b9e1f41cc808a076c2fcf4409 (127.20.178.65:32925)
I20260929 03:17:37.784050 22308 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.178.65@KRBTEST.COM'} at 127.20.178.65:51197
WARNING: no policy specified for kudu/127.20.178.66@KRBTEST.COM; defaulting to no policy
I20260929 03:17:37.794621 22358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:37.786570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.65:50601 (local address 127.20.178.126:36403)
0929 03:17:37.786669 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:37.786671 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:37.786682 (+ 11us) server_negotiation.cc:407] Connection header received
0929 03:17:37.786703 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:37.786729 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:37.786742 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:37.787478 (+ 736us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:17:37.787973 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:37.788771 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:37.789932 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:37.790234 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:37.790347 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:37.792004 (+ 1657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:17:37.792006 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:17:37.792007 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:17:37.792014 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:17:37.793696 (+ 1682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:37.793987 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:37.793988 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:37.793988 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:37.794032 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:37.794182 (+ 150us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:37.794182 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:37.794183 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:37.794267 (+ 84us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:17:37.794310 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0929 03:17:37.794419 (+ 109us) server_negotiation.cc:299] Negotiation successful
0929 03:17:37.794523 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260929 03:17:37.794632 22517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:37.784474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:36403 (local address 127.20.178.65:50601)
0929 03:17:37.784926 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0929 03:17:37.784939 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 03:17:37.785008 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:17:37.787510 (+ 2502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:17:37.787511 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:17:37.787782 (+ 271us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:17:37.787868 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:37.787870 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:37.788893 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:37.788894 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:37.789836 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:37.789839 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:37.790814 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:37.790815 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:37.790904 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:37.791456 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:17:37.791461 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:17:37.791863 (+ 402us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:17:37.793754 (+ 1891us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:17:37.793755 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:17:37.793755 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:17:37.793922 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:17:37.794087 (+ 165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:17:37.794087 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:17:37.794087 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:17:37.794140 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:17:37.794316 (+ 176us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:17:37.794317 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:17:37.794352 (+ 35us) client_negotiation.cc:769] Sending connection context
0929 03:17:37.794403 (+ 51us) client_negotiation.cc:240] Negotiation successful
0929 03:17:37.794476 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":72,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.178.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.178.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.66.keytab.
Entry for principal kudu/127.20.178.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.66.keytab.
I20260929 03:17:37.811153 21193 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuoaHbX/build/release/bin/kudu
/tmp/dist-test-taskuoaHbX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651852476208-21193-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651852476208-21193-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651852476208-21193-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651852476208-21193-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.178.66:0
--local_ip_for_outbound_sockets=127.20.178.66
--webserver_interface=127.20.178.66
--webserver_port=0
--tserver_master_addrs=127.20.178.126:36403
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.66.keytab
--principal=kudu/127.20.178.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.178.84:38049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651852476208-21193-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651852476208-21193-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651852476208-21193-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:17:37.890115 22520 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:17:37.890380 22520 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:17:37.890419 22520 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:17:37.891589 22520 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:17:37.891686 22520 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:17:37.891723 22520 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 03:17:37.891762 22520 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.178.66
I20260929 03:17:37.893219 22520 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.178.84:38049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651852476208-21193-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651852476208-21193-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.66.keytab
--principal=kudu/127.20.178.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.178.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651852476208-21193-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.178.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.20.178.126:36403
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22520
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.178.66
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651852476208-21193-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:17:37.893488 22520 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:17:37.893725 22520 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:17:37.898909 22526 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:17:37.898983 22525 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:17:37.899000 22520 server_base.cc:1034] running on GCE node
W20260929 03:17:37.898909 22528 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:17:37.899281 22520 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:17:37.899523 22520 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:17:37.900695 22520 hybrid_clock.cc:648] HybridClock initialized: now 1790651857900682 us; error 29 us; skew 500 ppm
Sep 29 03:17:37 dist-test-slave-c9qh krb5kdc[22263](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651857, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:17:37.903326 22520 init.cc:377] Logged in from keytab as kudu/127.20.178.66@KRBTEST.COM (short username kudu)
I20260929 03:17:37.903805 22520 webserver.cc:460] Webserver started at http://127.20.178.66:38253/ using document root <none> and password file <none>
I20260929 03:17:37.904029 22520 fs_manager.cc:359] Metadata directory not provided
I20260929 03:17:37.904072 22520 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:17:37.904241 22520 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:17:37.905344 22520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651852476208-21193-0/minicluster-data/ts-1/data/instance:
uuid: "5fb0d1e69eba4b7998056ad25fcbc335"
format_stamp: "Formatted at 2026-09-29 03:17:37 on dist-test-slave-c9qh"
server_key: "6638b498a2d55c39b604b2ae277b9fc0"
server_key_iv: "3fbcd6e1097bde10c5d27c47e191e3bd"
server_key_version: "encryptionkey@0"
I20260929 03:17:37.905691 22520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651852476208-21193-0/minicluster-data/ts-1/wal/instance:
uuid: "5fb0d1e69eba4b7998056ad25fcbc335"
format_stamp: "Formatted at 2026-09-29 03:17:37 on dist-test-slave-c9qh"
server_key: "6638b498a2d55c39b604b2ae277b9fc0"
server_key_iv: "3fbcd6e1097bde10c5d27c47e191e3bd"
server_key_version: "encryptionkey@0"
I20260929 03:17:37.907145 22520 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 03:17:37.907809 22535 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:17:37.907998 22520 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 03:17:37.908085 22520 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651852476208-21193-0/minicluster-data/ts-1/data,/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651852476208-21193-0/minicluster-data/ts-1/wal
uuid: "5fb0d1e69eba4b7998056ad25fcbc335"
format_stamp: "Formatted at 2026-09-29 03:17:37 on dist-test-slave-c9qh"
server_key: "6638b498a2d55c39b604b2ae277b9fc0"
server_key_iv: "3fbcd6e1097bde10c5d27c47e191e3bd"
server_key_version: "encryptionkey@0"
I20260929 03:17:37.908162 22520 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651852476208-21193-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651852476208-21193-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651852476208-21193-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:17:37.929683 22520 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:17:37.932030 22520 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:17:37.932150 22520 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:17:37.932789 22520 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:17:37.932840 22520 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:17:37.932878 22520 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:17:37.932895 22520 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 03:17:37 dist-test-slave-c9qh krb5kdc[22263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651857, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.66@KRBTEST.COM for kudu/127.20.178.126@KRBTEST.COM
I20260929 03:17:37.940408 22520 rpc_server.cc:307] RPC server started. Bound to: 127.20.178.66:36871
I20260929 03:17:37.940490 22650 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.178.66:36871 every 8 connection(s)
I20260929 03:17:37.940809 22520 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651852476208-21193-0/minicluster-data/ts-1/data/info.pb
I20260929 03:17:37.942929 22358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:37.933201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.66:56317 (local address 127.20.178.126:36403)
0929 03:17:37.933272 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:37.933275 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:37.933577 (+ 302us) server_negotiation.cc:407] Connection header received
0929 03:17:37.934290 (+ 713us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:37.934290 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:37.934300 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:37.934383 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:17:37.935313 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:37.935909 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:37.937157 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:37.937354 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:37.937430 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:37.939931 (+ 2501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:17:37.939933 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:17:37.939934 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:17:37.939942 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:17:37.941555 (+ 1613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:37.941960 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:37.941961 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:37.941962 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:37.942026 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:37.942276 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:37.942277 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:37.942277 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:37.942399 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:17:37.942476 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0929 03:17:37.942753 (+ 277us) server_negotiation.cc:299] Negotiation successful
0929 03:17:37.942829 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260929 03:17:37.943147 22544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:37.933201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:36403 (local address 127.20.178.66:56317)
0929 03:17:37.933415 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0929 03:17:37.933469 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0929 03:17:37.934181 (+ 712us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:17:37.934484 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:17:37.934489 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:17:37.934899 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:17:37.935214 (+ 315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:37.935222 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:37.936013 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:37.936014 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:37.937058 (+ 1044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:37.937061 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:37.937458 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:37.937460 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:37.937589 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:37.938140 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:17:37.938151 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:17:37.939831 (+ 1680us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:17:37.941649 (+ 1818us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:17:37.941655 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:17:37.941657 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:17:37.941881 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:17:37.942093 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:17:37.942094 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:17:37.942095 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:17:37.942206 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:17:37.942467 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:17:37.942474 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:17:37.942621 (+ 147us) client_negotiation.cc:769] Sending connection context
0929 03:17:37.942739 (+ 118us) client_negotiation.cc:240] Negotiation successful
0929 03:17:37.942921 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":91,"threads_started":1}
I20260929 03:17:37.943609 22651 heartbeater.cc:347] Connected to a master server at 127.20.178.126:36403
I20260929 03:17:37.943689 22651 heartbeater.cc:464] Registering TS with master...
I20260929 03:17:37.943874 22651 heartbeater.cc:511] Master 127.20.178.126:36403 requested a full tablet report, sending...
I20260929 03:17:37.944242 22307 ts_manager.cc:194] Registered new tserver with Master: 5fb0d1e69eba4b7998056ad25fcbc335 (127.20.178.66:36871)
I20260929 03:17:37.944849 22307 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.178.66@KRBTEST.COM'} at 127.20.178.66:56317
I20260929 03:17:37.945756 21193 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuoaHbX/build/release/bin/kudu as pid 22520
I20260929 03:17:37.945842 21193 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651852476208-21193-0/minicluster-data/ts-1/wal/instance
I20260929 03:17:37.946013 21193 external_mini_cluster.cc:1428] Setting key 4c129eb288ff76139c2e98840d51b5ea
I20260929 03:17:37.950748 22358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:37.944708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.66:56607 (local address 127.20.178.126:36403)
0929 03:17:37.945017 (+ 309us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:37.945019 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:37.945027 (+ 8us) server_negotiation.cc:407] Connection header received
0929 03:17:37.945044 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:37.945044 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:37.945053 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:37.945109 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:17:37.945455 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:37.946135 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:37.946930 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:37.947133 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:37.947214 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:37.948266 (+ 1052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:17:37.948267 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:17:37.948268 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:17:37.948273 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:17:37.949548 (+ 1275us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:37.949996 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:37.949997 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:37.949998 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:37.950060 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:37.950276 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:37.950277 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:37.950277 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:37.950403 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:17:37.950455 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0929 03:17:37.950639 (+ 184us) server_negotiation.cc:299] Negotiation successful
0929 03:17:37.950692 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":291}
I20260929 03:17:37.950889 22657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:37.944590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:36403 (local address 127.20.178.66:56607)
0929 03:17:37.944765 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0929 03:17:37.944776 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 03:17:37.944827 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:17:37.945132 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:17:37.945133 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:17:37.945308 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:17:37.945368 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:37.945370 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:37.946214 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:37.946215 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:37.946831 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:37.946833 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:37.947216 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:37.947217 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:37.947290 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:37.947747 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:17:37.947753 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:17:37.948178 (+ 425us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:17:37.949626 (+ 1448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:17:37.949628 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:17:37.949628 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:17:37.949900 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:17:37.950136 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:17:37.950137 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:17:37.950137 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:17:37.950212 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:17:37.950470 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:17:37.950471 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:17:37.950523 (+ 52us) client_negotiation.cc:769] Sending connection context
0929 03:17:37.950619 (+ 96us) client_negotiation.cc:240] Negotiation successful
0929 03:17:37.950739 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":108,"threads_started":1}
WARNING: no policy specified for kudu/127.20.178.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.178.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.178.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.67.keytab.
Entry for principal kudu/127.20.178.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.67.keytab.
I20260929 03:17:37.971061 21193 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuoaHbX/build/release/bin/kudu
/tmp/dist-test-taskuoaHbX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651852476208-21193-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651852476208-21193-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651852476208-21193-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651852476208-21193-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.178.67:0
--local_ip_for_outbound_sockets=127.20.178.67
--webserver_interface=127.20.178.67
--webserver_port=0
--tserver_master_addrs=127.20.178.126:36403
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.67.keytab
--principal=kudu/127.20.178.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.178.84:38049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651852476208-21193-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651852476208-21193-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651852476208-21193-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:17:38.050648 22660 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:17:38.050854 22660 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:17:38.050877 22660 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:17:38.052100 22660 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:17:38.052148 22660 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:17:38.052168 22660 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 03:17:38.052186 22660 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.178.67
I20260929 03:17:38.053687 22660 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.178.84:38049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651852476208-21193-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651852476208-21193-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.67.keytab
--principal=kudu/127.20.178.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.178.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651852476208-21193-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.178.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.20.178.126:36403
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22660
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.178.67
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651852476208-21193-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:17:38.053952 22660 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:17:38.054189 22660 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:17:38.059239 22668 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:17:38.059327 22666 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:17:38.059281 22660 server_base.cc:1034] running on GCE node
W20260929 03:17:38.059450 22665 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:17:38.059669 22660 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:17:38.059942 22660 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:17:38.061105 22660 hybrid_clock.cc:648] HybridClock initialized: now 1790651858061099 us; error 40 us; skew 500 ppm
Sep 29 03:17:38 dist-test-slave-c9qh krb5kdc[22263](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651858, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:17:38.063578 22660 init.cc:377] Logged in from keytab as kudu/127.20.178.67@KRBTEST.COM (short username kudu)
I20260929 03:17:38.064067 22660 webserver.cc:460] Webserver started at http://127.20.178.67:35087/ using document root <none> and password file <none>
I20260929 03:17:38.064317 22660 fs_manager.cc:359] Metadata directory not provided
I20260929 03:17:38.064388 22660 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:17:38.064532 22660 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:17:38.065636 22660 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651852476208-21193-0/minicluster-data/ts-2/data/instance:
uuid: "4faca80c18564946aa198a96659d1cf5"
format_stamp: "Formatted at 2026-09-29 03:17:38 on dist-test-slave-c9qh"
server_key: "6180f40e7924077c2a19c856104093ab"
server_key_iv: "6409d54749f016ce139b35a5ef71ab4c"
server_key_version: "encryptionkey@0"
I20260929 03:17:38.066018 22660 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651852476208-21193-0/minicluster-data/ts-2/wal/instance:
uuid: "4faca80c18564946aa198a96659d1cf5"
format_stamp: "Formatted at 2026-09-29 03:17:38 on dist-test-slave-c9qh"
server_key: "6180f40e7924077c2a19c856104093ab"
server_key_iv: "6409d54749f016ce139b35a5ef71ab4c"
server_key_version: "encryptionkey@0"
I20260929 03:17:38.067389 22660 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 03:17:38.068114 22675 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:17:38.068289 22660 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 03:17:38.068394 22660 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651852476208-21193-0/minicluster-data/ts-2/data,/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651852476208-21193-0/minicluster-data/ts-2/wal
uuid: "4faca80c18564946aa198a96659d1cf5"
format_stamp: "Formatted at 2026-09-29 03:17:38 on dist-test-slave-c9qh"
server_key: "6180f40e7924077c2a19c856104093ab"
server_key_iv: "6409d54749f016ce139b35a5ef71ab4c"
server_key_version: "encryptionkey@0"
I20260929 03:17:38.068468 22660 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651852476208-21193-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651852476208-21193-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651852476208-21193-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:17:38.084398 22660 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:17:38.086318 22660 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:17:38.086493 22660 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:17:38.087076 22660 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:17:38.087138 22660 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:17:38.087183 22660 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:17:38.087205 22660 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 03:17:38 dist-test-slave-c9qh krb5kdc[22263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651858, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.67@KRBTEST.COM for kudu/127.20.178.126@KRBTEST.COM
I20260929 03:17:38.094740 22660 rpc_server.cc:307] RPC server started. Bound to: 127.20.178.67:37711
I20260929 03:17:38.094791 22790 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.178.67:37711 every 8 connection(s)
I20260929 03:17:38.095191 22660 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651852476208-21193-0/minicluster-data/ts-2/data/info.pb
I20260929 03:17:38.095793 21193 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuoaHbX/build/release/bin/kudu as pid 22660
I20260929 03:17:38.095888 21193 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651852476208-21193-0/minicluster-data/ts-2/wal/instance
I20260929 03:17:38.096078 21193 external_mini_cluster.cc:1428] Setting key 4baade24530e2d560033e27c3a6ab981
I20260929 03:17:38.096457 22358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:38.087494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.67:35819 (local address 127.20.178.126:36403)
0929 03:17:38.087568 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:38.087571 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:38.087787 (+ 216us) server_negotiation.cc:407] Connection header received
0929 03:17:38.088293 (+ 506us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:38.088294 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:38.088309 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:38.088395 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:17:38.089079 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:38.089572 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:38.090653 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:38.090856 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:38.090942 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:38.093301 (+ 2359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:17:38.093302 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:17:38.093302 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:17:38.093310 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:17:38.094941 (+ 1631us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:38.095381 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:38.095383 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:38.095383 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:38.095453 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:38.095751 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:38.095752 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:38.095753 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:38.095911 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:17:38.095992 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0929 03:17:38.096259 (+ 267us) server_negotiation.cc:299] Negotiation successful
0929 03:17:38.096356 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260929 03:17:38.096689 22684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:38.087481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:36403 (local address 127.20.178.67:35819)
0929 03:17:38.087647 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0929 03:17:38.087689 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 03:17:38.088187 (+ 498us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:17:38.088448 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:17:38.088451 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:17:38.088766 (+ 315us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:17:38.088982 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:38.088986 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:38.089688 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:38.089689 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:38.090553 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:38.090555 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:38.090952 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:38.090953 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:38.091069 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:38.091550 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:17:38.091558 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:17:38.093210 (+ 1652us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:17:38.095031 (+ 1821us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:17:38.095037 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:17:38.095039 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:17:38.095319 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:17:38.095542 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:17:38.095544 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:17:38.095544 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:17:38.095675 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:17:38.095983 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:17:38.095987 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:17:38.096106 (+ 119us) client_negotiation.cc:769] Sending connection context
0929 03:17:38.096246 (+ 140us) client_negotiation.cc:240] Negotiation successful
0929 03:17:38.096420 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":103,"threads_started":1}
I20260929 03:17:38.097204 22791 heartbeater.cc:347] Connected to a master server at 127.20.178.126:36403
I20260929 03:17:38.097308 22791 heartbeater.cc:464] Registering TS with master...
I20260929 03:17:38.097472 22791 heartbeater.cc:511] Master 127.20.178.126:36403 requested a full tablet report, sending...
I20260929 03:17:38.098266 22306 ts_manager.cc:194] Registered new tserver with Master: 4faca80c18564946aa198a96659d1cf5 (127.20.178.67:37711)
I20260929 03:17:38.098786 22306 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.178.67@KRBTEST.COM'} at 127.20.178.67:35819
I20260929 03:17:38.103399 22358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:38.098375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.67:45423 (local address 127.20.178.126:36403)
0929 03:17:38.098426 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:38.098428 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:38.098488 (+ 60us) server_negotiation.cc:407] Connection header received
0929 03:17:38.098550 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:38.098551 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:38.098560 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:38.098614 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:17:38.098960 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:38.099472 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:38.100129 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:38.100312 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:38.100363 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:38.101177 (+ 814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:17:38.101178 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:17:38.101178 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:17:38.101184 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:17:38.102577 (+ 1393us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:38.102860 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:38.102860 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:38.102861 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:38.102907 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:38.103047 (+ 140us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:38.103047 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:38.103047 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:38.103126 (+ 79us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:17:38.103195 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0929 03:17:38.103296 (+ 101us) server_negotiation.cc:299] Negotiation successful
0929 03:17:38.103345 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260929 03:17:38.103431 22797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:38.098322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:36403 (local address 127.20.178.67:45423)
0929 03:17:38.098432 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0929 03:17:38.098442 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 03:17:38.098499 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:17:38.098617 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:17:38.098617 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:17:38.098805 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:17:38.098883 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:38.098885 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:38.099571 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:38.099572 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:38.100058 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:38.100059 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:38.100378 (+ 319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:38.100378 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:38.100430 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:38.100829 (+ 399us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:17:38.100833 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:17:38.101098 (+ 265us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:17:38.102645 (+ 1547us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:17:38.102645 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:17:38.102646 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:17:38.102800 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:17:38.102963 (+ 163us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:17:38.102964 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:17:38.102964 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:17:38.103010 (+ 46us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:17:38.103194 (+ 184us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:17:38.103195 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:17:38.103225 (+ 30us) client_negotiation.cc:769] Sending connection context
0929 03:17:38.103281 (+ 56us) client_negotiation.cc:240] Negotiation successful
0929 03:17:38.103348 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":85,"thread_start_us":63,"threads_started":1}
I20260929 03:17:38.107893 21193 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 03:17:38 dist-test-slave-c9qh krb5kdc[22263](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651858, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 03:17:38 dist-test-slave-c9qh krb5kdc[22263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651858, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.20.178.126@KRBTEST.COM
I20260929 03:17:38.129616 22358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:38.122941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37942 (local address 127.20.178.126:36403)
0929 03:17:38.123004 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:38.123006 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:38.123033 (+ 27us) server_negotiation.cc:407] Connection header received
0929 03:17:38.123054 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:38.123055 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:38.123064 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:38.123104 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:17:38.123677 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:38.124154 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:38.124905 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:38.125087 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:38.125142 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:38.127354 (+ 2212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:17:38.127355 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:17:38.127356 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:17:38.127361 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:17:38.128565 (+ 1204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:38.128908 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:38.128909 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:38.128909 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:38.128961 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:38.129136 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:38.129137 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:38.129137 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:38.129232 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:17:38.129317 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0929 03:17:38.129407 (+ 90us) server_negotiation.cc:299] Negotiation successful
0929 03:17:38.129460 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260929 03:17:38.131336 22306 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37942:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 03:17:38.133088 22306 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 03:17:38.139663 22809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:38.135342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.65:32925 (local address 127.0.0.1:33346)
0929 03:17:38.135497 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0929 03:17:38.135542 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 03:17:38.135615 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:17:38.135915 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:17:38.135919 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:17:38.135921 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:17:38.135983 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:38.135985 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:38.137173 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:38.137174 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:38.138421 (+ 1247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:38.138424 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:38.139332 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:38.139333 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:38.139421 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:38.139435 (+ 14us) client_negotiation.cc:769] Sending connection context
0929 03:17:38.139509 (+ 74us) client_negotiation.cc:240] Negotiation successful
0929 03:17:38.139562 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":60,"threads_started":1}
I20260929 03:17:38.139782 22808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:38.135370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.66:36871 (local address 127.0.0.1:39168)
0929 03:17:38.135644 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0929 03:17:38.135655 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 03:17:38.135706 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:17:38.136308 (+ 602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:17:38.136309 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:17:38.136311 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:17:38.136341 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:38.136342 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:38.137623 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:38.137624 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:38.138622 (+ 998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:38.138625 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:38.139526 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:38.139527 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:38.139578 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:38.139581 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 03:17:38.139640 (+ 59us) client_negotiation.cc:240] Negotiation successful
0929 03:17:38.139682 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"mutex_wait_us":8,"thread_start_us":75,"threads_started":1}
I20260929 03:17:38.139813 22812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:38.135525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39168 (local address 127.20.178.66:36871)
0929 03:17:38.136133 (+ 608us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:38.136137 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:38.136159 (+ 22us) server_negotiation.cc:407] Connection header received
0929 03:17:38.136194 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:38.136196 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:38.136211 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:38.136414 (+ 203us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:17:38.136471 (+ 57us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:38.137528 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:38.138734 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:38.139431 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:38.139520 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:38.139571 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0929 03:17:38.139651 (+ 80us) server_negotiation.cc:299] Negotiation successful
0929 03:17:38.139689 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":571,"thread_start_us":84,"threads_started":1}
I20260929 03:17:38.140069 22811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:38.135533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.67:37711 (local address 127.0.0.1:59232)
0929 03:17:38.135912 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0929 03:17:38.135921 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 03:17:38.135973 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:17:38.136207 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:17:38.136208 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:17:38.136210 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:17:38.136257 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:38.136259 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:38.137609 (+ 1350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:38.137610 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:38.138654 (+ 1044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:38.138656 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:38.139864 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:38.139865 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:38.139924 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:38.139928 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 03:17:38.139984 (+ 56us) client_negotiation.cc:240] Negotiation successful
0929 03:17:38.140017 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":361,"mutex_wait_us":29,"thread_start_us":65,"threads_started":1}
I20260929 03:17:38.140146 22813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:38.135721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59232 (local address 127.20.178.67:37711)
0929 03:17:38.135892 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:38.135897 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:38.135968 (+ 71us) server_negotiation.cc:407] Connection header received
0929 03:17:38.136037 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:38.136039 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:38.136054 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:38.136142 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:17:38.136335 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:38.137509 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:38.138735 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:38.139245 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:38.139341 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:38.139399 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0929 03:17:38.140005 (+ 606us) server_negotiation.cc:299] Negotiation successful
0929 03:17:38.140050 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":107,"threads_started":1}
I20260929 03:17:38.139700 22810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:38.135442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33346 (local address 127.20.178.65:32925)
0929 03:17:38.135731 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:38.135735 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:38.135744 (+ 9us) server_negotiation.cc:407] Connection header received
0929 03:17:38.135787 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:38.135789 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:38.135802 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:38.135908 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:17:38.136213 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:38.137083 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:38.138505 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:38.139237 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:38.139313 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:38.139369 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0929 03:17:38.139539 (+ 170us) server_negotiation.cc:299] Negotiation successful
0929 03:17:38.139595 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":251,"thread_start_us":72,"threads_started":1}
I20260929 03:17:38.140829 22725 tablet_service.cc:1467] Processing CreateTablet for tablet c31f0b82d9464d208562da885a8d09d9 (DEFAULT_TABLE table=test-table [id=d2fb15fee52a4aa6b04a6023a21581d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:17:38.141065 22445 tablet_service.cc:1467] Processing CreateTablet for tablet c31f0b82d9464d208562da885a8d09d9 (DEFAULT_TABLE table=test-table [id=d2fb15fee52a4aa6b04a6023a21581d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:17:38.141206 22725 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c31f0b82d9464d208562da885a8d09d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:17:38.141394 22445 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c31f0b82d9464d208562da885a8d09d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:17:38.143893 22814 tablet_bootstrap.cc:492] T c31f0b82d9464d208562da885a8d09d9 P 4faca80c18564946aa198a96659d1cf5: Bootstrap starting.
I20260929 03:17:38.144279 22585 tablet_service.cc:1467] Processing CreateTablet for tablet c31f0b82d9464d208562da885a8d09d9 (DEFAULT_TABLE table=test-table [id=d2fb15fee52a4aa6b04a6023a21581d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:17:38.144451 22815 tablet_bootstrap.cc:492] T c31f0b82d9464d208562da885a8d09d9 P 64ec142b9e1f41cc808a076c2fcf4409: Bootstrap starting.
I20260929 03:17:38.144609 22585 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c31f0b82d9464d208562da885a8d09d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:17:38.144991 22814 tablet_bootstrap.cc:654] T c31f0b82d9464d208562da885a8d09d9 P 4faca80c18564946aa198a96659d1cf5: Neither blocks nor log segments found. Creating new log.
I20260929 03:17:38.145128 22815 tablet_bootstrap.cc:654] T c31f0b82d9464d208562da885a8d09d9 P 64ec142b9e1f41cc808a076c2fcf4409: Neither blocks nor log segments found. Creating new log.
I20260929 03:17:38.145363 22814 log.cc:824] T c31f0b82d9464d208562da885a8d09d9 P 4faca80c18564946aa198a96659d1cf5: Log is configured to *not* fsync() on all Append() calls
I20260929 03:17:38.145569 22815 log.cc:824] T c31f0b82d9464d208562da885a8d09d9 P 64ec142b9e1f41cc808a076c2fcf4409: Log is configured to *not* fsync() on all Append() calls
I20260929 03:17:38.146056 22814 tablet_bootstrap.cc:492] T c31f0b82d9464d208562da885a8d09d9 P 4faca80c18564946aa198a96659d1cf5: No bootstrap required, opened a new log
I20260929 03:17:38.146112 22814 ts_tablet_manager.cc:1397] T c31f0b82d9464d208562da885a8d09d9 P 4faca80c18564946aa198a96659d1cf5: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 03:17:38.147720 22814 raft_consensus.cc:348] T c31f0b82d9464d208562da885a8d09d9 P 4faca80c18564946aa198a96659d1cf5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ec142b9e1f41cc808a076c2fcf4409" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 32925 } } peers { permanent_uuid: "4faca80c18564946aa198a96659d1cf5" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 37711 } } peers { permanent_uuid: "5fb0d1e69eba4b7998056ad25fcbc335" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 36871 } }
I20260929 03:17:38.147869 22814 raft_consensus.cc:374] T c31f0b82d9464d208562da885a8d09d9 P 4faca80c18564946aa198a96659d1cf5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:17:38.147910 22814 raft_consensus.cc:729] T c31f0b82d9464d208562da885a8d09d9 P 4faca80c18564946aa198a96659d1cf5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4faca80c18564946aa198a96659d1cf5, State: Initialized, Role: FOLLOWER
I20260929 03:17:38.147921 22818 tablet_bootstrap.cc:492] T c31f0b82d9464d208562da885a8d09d9 P 5fb0d1e69eba4b7998056ad25fcbc335: Bootstrap starting.
I20260929 03:17:38.148054 22814 consensus_queue.cc:260] T c31f0b82d9464d208562da885a8d09d9 P 4faca80c18564946aa198a96659d1cf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ec142b9e1f41cc808a076c2fcf4409" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 32925 } } peers { permanent_uuid: "4faca80c18564946aa198a96659d1cf5" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 37711 } } peers { permanent_uuid: "5fb0d1e69eba4b7998056ad25fcbc335" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 36871 } }
I20260929 03:17:38.148320 22814 ts_tablet_manager.cc:1428] T c31f0b82d9464d208562da885a8d09d9 P 4faca80c18564946aa198a96659d1cf5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 03:17:38.148428 22791 heartbeater.cc:503] Master 127.20.178.126:36403 was elected leader, sending a full tablet report...
I20260929 03:17:38.148532 22818 tablet_bootstrap.cc:654] T c31f0b82d9464d208562da885a8d09d9 P 5fb0d1e69eba4b7998056ad25fcbc335: Neither blocks nor log segments found. Creating new log.
I20260929 03:17:38.148902 22818 log.cc:824] T c31f0b82d9464d208562da885a8d09d9 P 5fb0d1e69eba4b7998056ad25fcbc335: Log is configured to *not* fsync() on all Append() calls
I20260929 03:17:38.149003 22815 tablet_bootstrap.cc:492] T c31f0b82d9464d208562da885a8d09d9 P 64ec142b9e1f41cc808a076c2fcf4409: No bootstrap required, opened a new log
I20260929 03:17:38.149072 22815 ts_tablet_manager.cc:1397] T c31f0b82d9464d208562da885a8d09d9 P 64ec142b9e1f41cc808a076c2fcf4409: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260929 03:17:38.149562 22818 tablet_bootstrap.cc:492] T c31f0b82d9464d208562da885a8d09d9 P 5fb0d1e69eba4b7998056ad25fcbc335: No bootstrap required, opened a new log
I20260929 03:17:38.149616 22818 ts_tablet_manager.cc:1397] T c31f0b82d9464d208562da885a8d09d9 P 5fb0d1e69eba4b7998056ad25fcbc335: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260929 03:17:38.150519 22815 raft_consensus.cc:348] T c31f0b82d9464d208562da885a8d09d9 P 64ec142b9e1f41cc808a076c2fcf4409 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ec142b9e1f41cc808a076c2fcf4409" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 32925 } } peers { permanent_uuid: "4faca80c18564946aa198a96659d1cf5" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 37711 } } peers { permanent_uuid: "5fb0d1e69eba4b7998056ad25fcbc335" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 36871 } }
I20260929 03:17:38.150664 22815 raft_consensus.cc:374] T c31f0b82d9464d208562da885a8d09d9 P 64ec142b9e1f41cc808a076c2fcf4409 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:17:38.150708 22815 raft_consensus.cc:729] T c31f0b82d9464d208562da885a8d09d9 P 64ec142b9e1f41cc808a076c2fcf4409 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64ec142b9e1f41cc808a076c2fcf4409, State: Initialized, Role: FOLLOWER
I20260929 03:17:38.150818 22815 consensus_queue.cc:260] T c31f0b82d9464d208562da885a8d09d9 P 64ec142b9e1f41cc808a076c2fcf4409 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ec142b9e1f41cc808a076c2fcf4409" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 32925 } } peers { permanent_uuid: "4faca80c18564946aa198a96659d1cf5" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 37711 } } peers { permanent_uuid: "5fb0d1e69eba4b7998056ad25fcbc335" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 36871 } }
I20260929 03:17:38.151064 22815 ts_tablet_manager.cc:1428] T c31f0b82d9464d208562da885a8d09d9 P 64ec142b9e1f41cc808a076c2fcf4409: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 03:17:38.150981 22818 raft_consensus.cc:348] T c31f0b82d9464d208562da885a8d09d9 P 5fb0d1e69eba4b7998056ad25fcbc335 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ec142b9e1f41cc808a076c2fcf4409" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 32925 } } peers { permanent_uuid: "4faca80c18564946aa198a96659d1cf5" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 37711 } } peers { permanent_uuid: "5fb0d1e69eba4b7998056ad25fcbc335" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 36871 } }
I20260929 03:17:38.151129 22511 heartbeater.cc:503] Master 127.20.178.126:36403 was elected leader, sending a full tablet report...
I20260929 03:17:38.151134 22818 raft_consensus.cc:374] T c31f0b82d9464d208562da885a8d09d9 P 5fb0d1e69eba4b7998056ad25fcbc335 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:17:38.151175 22818 raft_consensus.cc:729] T c31f0b82d9464d208562da885a8d09d9 P 5fb0d1e69eba4b7998056ad25fcbc335 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fb0d1e69eba4b7998056ad25fcbc335, State: Initialized, Role: FOLLOWER
I20260929 03:17:38.151292 22818 consensus_queue.cc:260] T c31f0b82d9464d208562da885a8d09d9 P 5fb0d1e69eba4b7998056ad25fcbc335 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ec142b9e1f41cc808a076c2fcf4409" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 32925 } } peers { permanent_uuid: "4faca80c18564946aa198a96659d1cf5" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 37711 } } peers { permanent_uuid: "5fb0d1e69eba4b7998056ad25fcbc335" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 36871 } }
I20260929 03:17:38.151551 22818 ts_tablet_manager.cc:1428] T c31f0b82d9464d208562da885a8d09d9 P 5fb0d1e69eba4b7998056ad25fcbc335: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260929 03:17:38.151592 22651 heartbeater.cc:503] Master 127.20.178.126:36403 was elected leader, sending a full tablet report...
W20260929 03:17:38.191476 22652 tablet.cc:2369] T c31f0b82d9464d208562da885a8d09d9 P 5fb0d1e69eba4b7998056ad25fcbc335: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 03:17:38.200495 22819 raft_consensus.cc:482] T c31f0b82d9464d208562da885a8d09d9 P 4faca80c18564946aa198a96659d1cf5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:17:38.200606 22819 raft_consensus.cc:504] T c31f0b82d9464d208562da885a8d09d9 P 4faca80c18564946aa198a96659d1cf5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ec142b9e1f41cc808a076c2fcf4409" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 32925 } } peers { permanent_uuid: "4faca80c18564946aa198a96659d1cf5" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 37711 } } peers { permanent_uuid: "5fb0d1e69eba4b7998056ad25fcbc335" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 36871 } }
I20260929 03:17:38.200977 22819 leader_election.cc:288] T c31f0b82d9464d208562da885a8d09d9 P 4faca80c18564946aa198a96659d1cf5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 64ec142b9e1f41cc808a076c2fcf4409 (127.20.178.65:32925), 5fb0d1e69eba4b7998056ad25fcbc335 (127.20.178.66:36871)
I20260929 03:17:38.204509 22684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:38.201155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.65:32925 (local address 127.20.178.67:59157)
0929 03:17:38.201235 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0929 03:17:38.201248 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 03:17:38.201325 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:17:38.201499 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:17:38.201500 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:17:38.201501 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:17:38.201606 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:38.201608 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:38.202555 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:38.202556 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:38.203653 (+ 1097us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:38.203655 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:38.204323 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:38.204323 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:38.204386 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:38.204389 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 03:17:38.204431 (+ 42us) client_negotiation.cc:240] Negotiation successful
0929 03:17:38.204458 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260929 03:17:38.204581 22810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:38.201313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.67:59157 (local address 127.20.178.65:32925)
0929 03:17:38.201359 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:38.201362 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:38.201371 (+ 9us) server_negotiation.cc:407] Connection header received
0929 03:17:38.201402 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:38.201403 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:38.201414 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:38.201484 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:17:38.201707 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:38.202473 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:38.203747 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:38.204242 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:38.204317 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:38.204324 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0929 03:17:38.204462 (+ 138us) server_negotiation.cc:299] Negotiation successful
0929 03:17:38.204499 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260929 03:17:38.204910 22812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:38.201367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.67:57493 (local address 127.20.178.66:36871)
0929 03:17:38.201420 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:38.201422 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:38.201799 (+ 377us) server_negotiation.cc:407] Connection header received
0929 03:17:38.201917 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:38.201918 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:38.201954 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:38.202029 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:17:38.202215 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:38.202807 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:38.203949 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:38.204496 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:38.204562 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:38.204569 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0929 03:17:38.204808 (+ 239us) server_negotiation.cc:299] Negotiation successful
0929 03:17:38.204857 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260929 03:17:38.204905 22465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c31f0b82d9464d208562da885a8d09d9" candidate_uuid: "4faca80c18564946aa198a96659d1cf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64ec142b9e1f41cc808a076c2fcf4409" is_pre_election: true
I20260929 03:17:38.205080 22465 raft_consensus.cc:2454] T c31f0b82d9464d208562da885a8d09d9 P 64ec142b9e1f41cc808a076c2fcf4409 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4faca80c18564946aa198a96659d1cf5 in term 0.
I20260929 03:17:38.205122 22823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:38.201156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.66:36871 (local address 127.20.178.67:57493)
0929 03:17:38.201733 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0929 03:17:38.201743 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 03:17:38.201828 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:17:38.202028 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:17:38.202029 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:17:38.202044 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:17:38.202131 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:38.202132 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:38.202884 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:38.202885 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:38.203870 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:38.203872 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:38.204595 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:38.204596 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:38.204709 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:38.204714 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 03:17:38.205006 (+ 292us) client_negotiation.cc:240] Negotiation successful
0929 03:17:38.205052 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":562,"mutex_wait_us":363,"thread_start_us":130,"threads_started":1}
I20260929 03:17:38.205327 22676 leader_election.cc:302] T c31f0b82d9464d208562da885a8d09d9 P 4faca80c18564946aa198a96659d1cf5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4faca80c18564946aa198a96659d1cf5, 64ec142b9e1f41cc808a076c2fcf4409; no voters:
I20260929 03:17:38.205453 22605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c31f0b82d9464d208562da885a8d09d9" candidate_uuid: "4faca80c18564946aa198a96659d1cf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fb0d1e69eba4b7998056ad25fcbc335" is_pre_election: true
I20260929 03:17:38.205497 22819 raft_consensus.cc:2781] T c31f0b82d9464d208562da885a8d09d9 P 4faca80c18564946aa198a96659d1cf5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 03:17:38.205576 22819 raft_consensus.cc:482] T c31f0b82d9464d208562da885a8d09d9 P 4faca80c18564946aa198a96659d1cf5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 03:17:38.205610 22819 raft_consensus.cc:3037] T c31f0b82d9464d208562da885a8d09d9 P 4faca80c18564946aa198a96659d1cf5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:17:38.205605 22605 raft_consensus.cc:2454] T c31f0b82d9464d208562da885a8d09d9 P 5fb0d1e69eba4b7998056ad25fcbc335 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4faca80c18564946aa198a96659d1cf5 in term 0.
I20260929 03:17:38.206550 22819 raft_consensus.cc:504] T c31f0b82d9464d208562da885a8d09d9 P 4faca80c18564946aa198a96659d1cf5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ec142b9e1f41cc808a076c2fcf4409" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 32925 } } peers { permanent_uuid: "4faca80c18564946aa198a96659d1cf5" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 37711 } } peers { permanent_uuid: "5fb0d1e69eba4b7998056ad25fcbc335" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 36871 } }
I20260929 03:17:38.206707 22819 leader_election.cc:288] T c31f0b82d9464d208562da885a8d09d9 P 4faca80c18564946aa198a96659d1cf5 [CANDIDATE]: Term 1 election: Requested vote from peers 64ec142b9e1f41cc808a076c2fcf4409 (127.20.178.65:32925), 5fb0d1e69eba4b7998056ad25fcbc335 (127.20.178.66:36871)
I20260929 03:17:38.206869 22605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c31f0b82d9464d208562da885a8d09d9" candidate_uuid: "4faca80c18564946aa198a96659d1cf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fb0d1e69eba4b7998056ad25fcbc335"
I20260929 03:17:38.206869 22465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c31f0b82d9464d208562da885a8d09d9" candidate_uuid: "4faca80c18564946aa198a96659d1cf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64ec142b9e1f41cc808a076c2fcf4409"
I20260929 03:17:38.206944 22605 raft_consensus.cc:3037] T c31f0b82d9464d208562da885a8d09d9 P 5fb0d1e69eba4b7998056ad25fcbc335 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:17:38.206945 22465 raft_consensus.cc:3037] T c31f0b82d9464d208562da885a8d09d9 P 64ec142b9e1f41cc808a076c2fcf4409 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:17:38.207788 22605 raft_consensus.cc:2454] T c31f0b82d9464d208562da885a8d09d9 P 5fb0d1e69eba4b7998056ad25fcbc335 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4faca80c18564946aa198a96659d1cf5 in term 1.
I20260929 03:17:38.207787 22465 raft_consensus.cc:2454] T c31f0b82d9464d208562da885a8d09d9 P 64ec142b9e1f41cc808a076c2fcf4409 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4faca80c18564946aa198a96659d1cf5 in term 1.
I20260929 03:17:38.207976 22676 leader_election.cc:302] T c31f0b82d9464d208562da885a8d09d9 P 4faca80c18564946aa198a96659d1cf5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4faca80c18564946aa198a96659d1cf5, 64ec142b9e1f41cc808a076c2fcf4409; no voters:
I20260929 03:17:38.208077 22819 raft_consensus.cc:2781] T c31f0b82d9464d208562da885a8d09d9 P 4faca80c18564946aa198a96659d1cf5 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:17:38.208253 22819 raft_consensus.cc:686] T c31f0b82d9464d208562da885a8d09d9 P 4faca80c18564946aa198a96659d1cf5 [term 1 LEADER]: Becoming Leader. State: Replica: 4faca80c18564946aa198a96659d1cf5, State: Running, Role: LEADER
I20260929 03:17:38.208374 22819 consensus_queue.cc:237] T c31f0b82d9464d208562da885a8d09d9 P 4faca80c18564946aa198a96659d1cf5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ec142b9e1f41cc808a076c2fcf4409" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 32925 } } peers { permanent_uuid: "4faca80c18564946aa198a96659d1cf5" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 37711 } } peers { permanent_uuid: "5fb0d1e69eba4b7998056ad25fcbc335" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 36871 } }
I20260929 03:17:38.209234 22308 catalog_manager.cc:5495] T c31f0b82d9464d208562da885a8d09d9 P 4faca80c18564946aa198a96659d1cf5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4faca80c18564946aa198a96659d1cf5 (127.20.178.67). New cstate: current_term: 1 leader_uuid: "4faca80c18564946aa198a96659d1cf5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ec142b9e1f41cc808a076c2fcf4409" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 32925 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4faca80c18564946aa198a96659d1cf5" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 37711 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5fb0d1e69eba4b7998056ad25fcbc335" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 36871 } health_report { overall_health: UNKNOWN } } }
I20260929 03:17:38.232265 22813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:38.229781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59238 (local address 127.20.178.67:37711)
0929 03:17:38.229844 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:38.229846 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:38.229855 (+ 9us) server_negotiation.cc:407] Connection header received
0929 03:17:38.229929 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:38.229931 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:38.229961 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:38.230034 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:17:38.230196 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:38.230656 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:38.231445 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:38.231637 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:38.231711 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:38.231925 (+ 214us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:17:38.232018 (+ 93us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:17:38.232105 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0929 03:17:38.232160 (+ 55us) server_negotiation.cc:299] Negotiation successful
0929 03:17:38.232180 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260929 03:17:38.233987 22465 raft_consensus.cc:1261] T c31f0b82d9464d208562da885a8d09d9 P 64ec142b9e1f41cc808a076c2fcf4409 [term 1 FOLLOWER]: Refusing update from remote peer 4faca80c18564946aa198a96659d1cf5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:17:38.234004 22605 raft_consensus.cc:1261] T c31f0b82d9464d208562da885a8d09d9 P 5fb0d1e69eba4b7998056ad25fcbc335 [term 1 FOLLOWER]: Refusing update from remote peer 4faca80c18564946aa198a96659d1cf5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:17:38.234360 22819 consensus_queue.cc:1035] T c31f0b82d9464d208562da885a8d09d9 P 4faca80c18564946aa198a96659d1cf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64ec142b9e1f41cc808a076c2fcf4409" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 32925 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:17:38.234511 22824 consensus_queue.cc:1035] T c31f0b82d9464d208562da885a8d09d9 P 4faca80c18564946aa198a96659d1cf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5fb0d1e69eba4b7998056ad25fcbc335" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 36871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:17:38.236812 22827 mvcc.cc:205] Tried to move back new op lower bound from 7334510011323871232 to 7334510011222290432. Current Snapshot: MvccSnapshot[applied={T|T < 7334510011323871232}]
I20260929 03:17:38.237787 22829 mvcc.cc:205] Tried to move back new op lower bound from 7334510011323871232 to 7334510011222290432. Current Snapshot: MvccSnapshot[applied={T|T < 7334510011323871232}]
I20260929 03:17:38.241039 22308 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37942:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:17:38.241137 22308 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37942:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:17:38.241897 22308 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 721722c2f2ff42e788510074c1a2d3fd: Sending DeleteTablet for 3 replicas of tablet c31f0b82d9464d208562da885a8d09d9
I20260929 03:17:38.242200 22445 tablet_service.cc:1514] Processing DeleteTablet for tablet c31f0b82d9464d208562da885a8d09d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:17:38 UTC) from {username='kudu', principal='kudu/127.20.178.126@KRBTEST.COM'} at 127.0.0.1:33346
I20260929 03:17:38.242225 22725 tablet_service.cc:1514] Processing DeleteTablet for tablet c31f0b82d9464d208562da885a8d09d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:17:38 UTC) from {username='kudu', principal='kudu/127.20.178.126@KRBTEST.COM'} at 127.0.0.1:59232
I20260929 03:17:38.242225 22585 tablet_service.cc:1514] Processing DeleteTablet for tablet c31f0b82d9464d208562da885a8d09d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:17:38 UTC) from {username='kudu', principal='kudu/127.20.178.126@KRBTEST.COM'} at 127.0.0.1:39168
I20260929 03:17:38.242465 22836 tablet_replica.cc:330] T c31f0b82d9464d208562da885a8d09d9 P 4faca80c18564946aa198a96659d1cf5: stopping tablet replica
I20260929 03:17:38.242487 22838 tablet_replica.cc:330] T c31f0b82d9464d208562da885a8d09d9 P 64ec142b9e1f41cc808a076c2fcf4409: stopping tablet replica
I20260929 03:17:38.242589 22836 raft_consensus.cc:2229] T c31f0b82d9464d208562da885a8d09d9 P 4faca80c18564946aa198a96659d1cf5 [term 1 LEADER]: Raft consensus shutting down.
I20260929 03:17:38.242595 22838 raft_consensus.cc:2229] T c31f0b82d9464d208562da885a8d09d9 P 64ec142b9e1f41cc808a076c2fcf4409 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:17:38.242704 22838 raft_consensus.cc:2258] T c31f0b82d9464d208562da885a8d09d9 P 64ec142b9e1f41cc808a076c2fcf4409 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:17:38.242789 22308 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37942:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 03:17:38.242949 22308 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 03:17:38.242884 22836 raft_consensus.cc:2258] T c31f0b82d9464d208562da885a8d09d9 P 4faca80c18564946aa198a96659d1cf5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:17:38.243142 22838 ts_tablet_manager.cc:1905] T c31f0b82d9464d208562da885a8d09d9 P 64ec142b9e1f41cc808a076c2fcf4409: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:17:38.243616 22836 ts_tablet_manager.cc:1905] T c31f0b82d9464d208562da885a8d09d9 P 4faca80c18564946aa198a96659d1cf5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:17:38.244372 22445 tablet_service.cc:1467] Processing CreateTablet for tablet 52d1395edcc64b50a8e4546a461dffc3 (DEFAULT_TABLE table=test-table [id=af9fd6429e7b4dc7aea0a7196588bc35]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:17:38.244379 22725 tablet_service.cc:1467] Processing CreateTablet for tablet 52d1395edcc64b50a8e4546a461dffc3 (DEFAULT_TABLE table=test-table [id=af9fd6429e7b4dc7aea0a7196588bc35]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:17:38.244410 22585 tablet_service.cc:1467] Processing CreateTablet for tablet 52d1395edcc64b50a8e4546a461dffc3 (DEFAULT_TABLE table=test-table [id=af9fd6429e7b4dc7aea0a7196588bc35]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:17:38.244485 22725 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 52d1395edcc64b50a8e4546a461dffc3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:17:38.244498 22445 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 52d1395edcc64b50a8e4546a461dffc3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:17:38.244513 22585 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 52d1395edcc64b50a8e4546a461dffc3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:17:38.245865 22814 tablet_bootstrap.cc:492] T 52d1395edcc64b50a8e4546a461dffc3 P 4faca80c18564946aa198a96659d1cf5: Bootstrap starting.
I20260929 03:17:38.246284 22814 tablet_bootstrap.cc:654] T 52d1395edcc64b50a8e4546a461dffc3 P 4faca80c18564946aa198a96659d1cf5: Neither blocks nor log segments found. Creating new log.
I20260929 03:17:38.246603 22818 tablet_bootstrap.cc:492] T 52d1395edcc64b50a8e4546a461dffc3 P 5fb0d1e69eba4b7998056ad25fcbc335: Bootstrap starting.
I20260929 03:17:38.246678 22815 tablet_bootstrap.cc:492] T 52d1395edcc64b50a8e4546a461dffc3 P 64ec142b9e1f41cc808a076c2fcf4409: Bootstrap starting.
I20260929 03:17:38.247009 22818 tablet_bootstrap.cc:654] T 52d1395edcc64b50a8e4546a461dffc3 P 5fb0d1e69eba4b7998056ad25fcbc335: Neither blocks nor log segments found. Creating new log.
I20260929 03:17:38.247085 22815 tablet_bootstrap.cc:654] T 52d1395edcc64b50a8e4546a461dffc3 P 64ec142b9e1f41cc808a076c2fcf4409: Neither blocks nor log segments found. Creating new log.
I20260929 03:17:38.247170 22838 ts_tablet_manager.cc:1918] T c31f0b82d9464d208562da885a8d09d9 P 64ec142b9e1f41cc808a076c2fcf4409: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:17:38.247257 22838 log.cc:1196] T c31f0b82d9464d208562da885a8d09d9 P 64ec142b9e1f41cc808a076c2fcf4409: Deleting WAL directory at /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651852476208-21193-0/minicluster-data/ts-0/wal/wals/c31f0b82d9464d208562da885a8d09d9
I20260929 03:17:38.247525 22838 ts_tablet_manager.cc:1939] T c31f0b82d9464d208562da885a8d09d9 P 64ec142b9e1f41cc808a076c2fcf4409: Deleting consensus metadata
I20260929 03:17:38.247975 22292 catalog_manager.cc:4841] TS 64ec142b9e1f41cc808a076c2fcf4409 (127.20.178.65:32925): tablet c31f0b82d9464d208562da885a8d09d9 (table test-table [id=d2fb15fee52a4aa6b04a6023a21581d2]) successfully deleted
I20260929 03:17:38.248208 22818 tablet_bootstrap.cc:492] T 52d1395edcc64b50a8e4546a461dffc3 P 5fb0d1e69eba4b7998056ad25fcbc335: No bootstrap required, opened a new log
I20260929 03:17:38.248275 22818 ts_tablet_manager.cc:1397] T 52d1395edcc64b50a8e4546a461dffc3 P 5fb0d1e69eba4b7998056ad25fcbc335: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260929 03:17:38.248355 22836 ts_tablet_manager.cc:1918] T c31f0b82d9464d208562da885a8d09d9 P 4faca80c18564946aa198a96659d1cf5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:17:38.248421 22836 log.cc:1196] T c31f0b82d9464d208562da885a8d09d9 P 4faca80c18564946aa198a96659d1cf5: Deleting WAL directory at /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651852476208-21193-0/minicluster-data/ts-2/wal/wals/c31f0b82d9464d208562da885a8d09d9
I20260929 03:17:38.248422 22818 raft_consensus.cc:348] T 52d1395edcc64b50a8e4546a461dffc3 P 5fb0d1e69eba4b7998056ad25fcbc335 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ec142b9e1f41cc808a076c2fcf4409" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 32925 } } peers { permanent_uuid: "5fb0d1e69eba4b7998056ad25fcbc335" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 36871 } } peers { permanent_uuid: "4faca80c18564946aa198a96659d1cf5" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 37711 } }
I20260929 03:17:38.248504 22818 raft_consensus.cc:374] T 52d1395edcc64b50a8e4546a461dffc3 P 5fb0d1e69eba4b7998056ad25fcbc335 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:17:38.248533 22818 raft_consensus.cc:729] T 52d1395edcc64b50a8e4546a461dffc3 P 5fb0d1e69eba4b7998056ad25fcbc335 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fb0d1e69eba4b7998056ad25fcbc335, State: Initialized, Role: FOLLOWER
I20260929 03:17:38.248579 22818 consensus_queue.cc:260] T 52d1395edcc64b50a8e4546a461dffc3 P 5fb0d1e69eba4b7998056ad25fcbc335 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ec142b9e1f41cc808a076c2fcf4409" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 32925 } } peers { permanent_uuid: "5fb0d1e69eba4b7998056ad25fcbc335" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 36871 } } peers { permanent_uuid: "4faca80c18564946aa198a96659d1cf5" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 37711 } }
I20260929 03:17:38.248701 22836 ts_tablet_manager.cc:1939] T c31f0b82d9464d208562da885a8d09d9 P 4faca80c18564946aa198a96659d1cf5: Deleting consensus metadata
I20260929 03:17:38.248698 22818 ts_tablet_manager.cc:1428] T 52d1395edcc64b50a8e4546a461dffc3 P 5fb0d1e69eba4b7998056ad25fcbc335: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260929 03:17:38.249101 22837 tablet_replica.cc:330] T c31f0b82d9464d208562da885a8d09d9 P 5fb0d1e69eba4b7998056ad25fcbc335: stopping tablet replica
I20260929 03:17:38.249138 22292 catalog_manager.cc:4841] TS 4faca80c18564946aa198a96659d1cf5 (127.20.178.67:37711): tablet c31f0b82d9464d208562da885a8d09d9 (table test-table [id=d2fb15fee52a4aa6b04a6023a21581d2]) successfully deleted
I20260929 03:17:38.249214 22837 raft_consensus.cc:2229] T c31f0b82d9464d208562da885a8d09d9 P 5fb0d1e69eba4b7998056ad25fcbc335 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:17:38.249323 22837 raft_consensus.cc:2258] T c31f0b82d9464d208562da885a8d09d9 P 5fb0d1e69eba4b7998056ad25fcbc335 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:17:38.249809 22814 tablet_bootstrap.cc:492] T 52d1395edcc64b50a8e4546a461dffc3 P 4faca80c18564946aa198a96659d1cf5: No bootstrap required, opened a new log
I20260929 03:17:38.249855 22814 ts_tablet_manager.cc:1397] T 52d1395edcc64b50a8e4546a461dffc3 P 4faca80c18564946aa198a96659d1cf5: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260929 03:17:38.250002 22814 raft_consensus.cc:348] T 52d1395edcc64b50a8e4546a461dffc3 P 4faca80c18564946aa198a96659d1cf5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ec142b9e1f41cc808a076c2fcf4409" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 32925 } } peers { permanent_uuid: "5fb0d1e69eba4b7998056ad25fcbc335" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 36871 } } peers { permanent_uuid: "4faca80c18564946aa198a96659d1cf5" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 37711 } }
I20260929 03:17:38.250083 22814 raft_consensus.cc:374] T 52d1395edcc64b50a8e4546a461dffc3 P 4faca80c18564946aa198a96659d1cf5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:17:38.250110 22814 raft_consensus.cc:729] T 52d1395edcc64b50a8e4546a461dffc3 P 4faca80c18564946aa198a96659d1cf5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4faca80c18564946aa198a96659d1cf5, State: Initialized, Role: FOLLOWER
I20260929 03:17:38.250159 22814 consensus_queue.cc:260] T 52d1395edcc64b50a8e4546a461dffc3 P 4faca80c18564946aa198a96659d1cf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ec142b9e1f41cc808a076c2fcf4409" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 32925 } } peers { permanent_uuid: "5fb0d1e69eba4b7998056ad25fcbc335" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 36871 } } peers { permanent_uuid: "4faca80c18564946aa198a96659d1cf5" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 37711 } }
I20260929 03:17:38.250247 22814 ts_tablet_manager.cc:1428] T 52d1395edcc64b50a8e4546a461dffc3 P 4faca80c18564946aa198a96659d1cf5: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260929 03:17:38.250500 22837 ts_tablet_manager.cc:1905] T c31f0b82d9464d208562da885a8d09d9 P 5fb0d1e69eba4b7998056ad25fcbc335: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:17:38.251901 22837 ts_tablet_manager.cc:1918] T c31f0b82d9464d208562da885a8d09d9 P 5fb0d1e69eba4b7998056ad25fcbc335: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:17:38.251964 22837 log.cc:1196] T c31f0b82d9464d208562da885a8d09d9 P 5fb0d1e69eba4b7998056ad25fcbc335: Deleting WAL directory at /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651852476208-21193-0/minicluster-data/ts-1/wal/wals/c31f0b82d9464d208562da885a8d09d9
I20260929 03:17:38.252214 22837 ts_tablet_manager.cc:1939] T c31f0b82d9464d208562da885a8d09d9 P 5fb0d1e69eba4b7998056ad25fcbc335: Deleting consensus metadata
I20260929 03:17:38.252616 22295 catalog_manager.cc:4841] TS 5fb0d1e69eba4b7998056ad25fcbc335 (127.20.178.66:36871): tablet c31f0b82d9464d208562da885a8d09d9 (table test-table [id=d2fb15fee52a4aa6b04a6023a21581d2]) successfully deleted
I20260929 03:17:38.255856 22815 tablet_bootstrap.cc:492] T 52d1395edcc64b50a8e4546a461dffc3 P 64ec142b9e1f41cc808a076c2fcf4409: No bootstrap required, opened a new log
I20260929 03:17:38.255924 22815 ts_tablet_manager.cc:1397] T 52d1395edcc64b50a8e4546a461dffc3 P 64ec142b9e1f41cc808a076c2fcf4409: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.000s
I20260929 03:17:38.256063 22815 raft_consensus.cc:348] T 52d1395edcc64b50a8e4546a461dffc3 P 64ec142b9e1f41cc808a076c2fcf4409 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ec142b9e1f41cc808a076c2fcf4409" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 32925 } } peers { permanent_uuid: "5fb0d1e69eba4b7998056ad25fcbc335" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 36871 } } peers { permanent_uuid: "4faca80c18564946aa198a96659d1cf5" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 37711 } }
I20260929 03:17:38.256146 22815 raft_consensus.cc:374] T 52d1395edcc64b50a8e4546a461dffc3 P 64ec142b9e1f41cc808a076c2fcf4409 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:17:38.256176 22815 raft_consensus.cc:729] T 52d1395edcc64b50a8e4546a461dffc3 P 64ec142b9e1f41cc808a076c2fcf4409 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64ec142b9e1f41cc808a076c2fcf4409, State: Initialized, Role: FOLLOWER
I20260929 03:17:38.256229 22815 consensus_queue.cc:260] T 52d1395edcc64b50a8e4546a461dffc3 P 64ec142b9e1f41cc808a076c2fcf4409 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ec142b9e1f41cc808a076c2fcf4409" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 32925 } } peers { permanent_uuid: "5fb0d1e69eba4b7998056ad25fcbc335" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 36871 } } peers { permanent_uuid: "4faca80c18564946aa198a96659d1cf5" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 37711 } }
I20260929 03:17:38.256317 22815 ts_tablet_manager.cc:1428] T 52d1395edcc64b50a8e4546a461dffc3 P 64ec142b9e1f41cc808a076c2fcf4409: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
W20260929 03:17:38.280303 22512 tablet.cc:2369] T 52d1395edcc64b50a8e4546a461dffc3 P 64ec142b9e1f41cc808a076c2fcf4409: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 03:17:38.327903 22821 raft_consensus.cc:482] T 52d1395edcc64b50a8e4546a461dffc3 P 64ec142b9e1f41cc808a076c2fcf4409 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:17:38.328032 22821 raft_consensus.cc:504] T 52d1395edcc64b50a8e4546a461dffc3 P 64ec142b9e1f41cc808a076c2fcf4409 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ec142b9e1f41cc808a076c2fcf4409" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 32925 } } peers { permanent_uuid: "5fb0d1e69eba4b7998056ad25fcbc335" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 36871 } } peers { permanent_uuid: "4faca80c18564946aa198a96659d1cf5" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 37711 } }
I20260929 03:17:38.328436 22821 leader_election.cc:288] T 52d1395edcc64b50a8e4546a461dffc3 P 64ec142b9e1f41cc808a076c2fcf4409 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5fb0d1e69eba4b7998056ad25fcbc335 (127.20.178.66:36871), 4faca80c18564946aa198a96659d1cf5 (127.20.178.67:37711)
I20260929 03:17:38.331746 22843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:38.328607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.67:37711 (local address 127.20.178.65:38633)
0929 03:17:38.328830 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0929 03:17:38.328841 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 03:17:38.328910 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:17:38.329271 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:17:38.329273 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:17:38.329276 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:17:38.329354 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:38.329356 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:38.330183 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:38.330184 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:38.330938 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:38.330940 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:38.331471 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:38.331472 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:38.331551 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:38.331554 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 03:17:38.331620 (+ 66us) client_negotiation.cc:240] Negotiation successful
0929 03:17:38.331659 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":207,"mutex_wait_us":2,"thread_start_us":80,"threads_started":1}
I20260929 03:17:38.331732 22813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:38.328749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.65:38633 (local address 127.20.178.67:37711)
0929 03:17:38.328795 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:38.328797 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:38.329118 (+ 321us) server_negotiation.cc:407] Connection header received
0929 03:17:38.329146 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:38.329147 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:38.329159 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:38.329212 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:17:38.329441 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:38.330081 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:38.331033 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:38.331388 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:38.331471 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:38.331480 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0929 03:17:38.331625 (+ 145us) server_negotiation.cc:299] Negotiation successful
0929 03:17:38.331665 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260929 03:17:38.332172 22745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52d1395edcc64b50a8e4546a461dffc3" candidate_uuid: "64ec142b9e1f41cc808a076c2fcf4409" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4faca80c18564946aa198a96659d1cf5" is_pre_election: true
I20260929 03:17:38.332304 22745 raft_consensus.cc:2454] T 52d1395edcc64b50a8e4546a461dffc3 P 4faca80c18564946aa198a96659d1cf5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 64ec142b9e1f41cc808a076c2fcf4409 in term 0.
I20260929 03:17:38.332412 22842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:38.328609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.66:36871 (local address 127.20.178.65:60685)
0929 03:17:38.328842 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0929 03:17:38.328850 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0929 03:17:38.328910 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:17:38.329089 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:17:38.329090 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:17:38.329092 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:17:38.329156 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:38.329158 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:38.330036 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:38.330037 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:38.331378 (+ 1341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:38.331381 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:38.332149 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:38.332150 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:38.332225 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:38.332229 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 03:17:38.332289 (+ 60us) client_negotiation.cc:240] Negotiation successful
0929 03:17:38.332331 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":79,"threads_started":1}
I20260929 03:17:38.332446 22812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:38.328753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.65:60685 (local address 127.20.178.66:36871)
0929 03:17:38.328798 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:38.328800 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:38.328936 (+ 136us) server_negotiation.cc:407] Connection header received
0929 03:17:38.328974 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:38.328976 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:38.328987 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:38.329076 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:17:38.329239 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:38.329934 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:38.331501 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:38.332071 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:38.332154 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:38.332163 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0929 03:17:38.332315 (+ 152us) server_negotiation.cc:299] Negotiation successful
0929 03:17:38.332363 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260929 03:17:38.332521 22396 leader_election.cc:302] T 52d1395edcc64b50a8e4546a461dffc3 P 64ec142b9e1f41cc808a076c2fcf4409 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4faca80c18564946aa198a96659d1cf5, 64ec142b9e1f41cc808a076c2fcf4409; no voters:
I20260929 03:17:38.332702 22821 raft_consensus.cc:2781] T 52d1395edcc64b50a8e4546a461dffc3 P 64ec142b9e1f41cc808a076c2fcf4409 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 03:17:38.332748 22821 raft_consensus.cc:482] T 52d1395edcc64b50a8e4546a461dffc3 P 64ec142b9e1f41cc808a076c2fcf4409 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 03:17:38.332782 22821 raft_consensus.cc:3037] T 52d1395edcc64b50a8e4546a461dffc3 P 64ec142b9e1f41cc808a076c2fcf4409 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:17:38.332816 22605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52d1395edcc64b50a8e4546a461dffc3" candidate_uuid: "64ec142b9e1f41cc808a076c2fcf4409" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fb0d1e69eba4b7998056ad25fcbc335" is_pre_election: true
I20260929 03:17:38.332897 22605 raft_consensus.cc:2454] T 52d1395edcc64b50a8e4546a461dffc3 P 5fb0d1e69eba4b7998056ad25fcbc335 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 64ec142b9e1f41cc808a076c2fcf4409 in term 0.
I20260929 03:17:38.333735 22821 raft_consensus.cc:504] T 52d1395edcc64b50a8e4546a461dffc3 P 64ec142b9e1f41cc808a076c2fcf4409 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ec142b9e1f41cc808a076c2fcf4409" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 32925 } } peers { permanent_uuid: "5fb0d1e69eba4b7998056ad25fcbc335" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 36871 } } peers { permanent_uuid: "4faca80c18564946aa198a96659d1cf5" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 37711 } }
I20260929 03:17:38.333899 22821 leader_election.cc:288] T 52d1395edcc64b50a8e4546a461dffc3 P 64ec142b9e1f41cc808a076c2fcf4409 [CANDIDATE]: Term 1 election: Requested vote from peers 5fb0d1e69eba4b7998056ad25fcbc335 (127.20.178.66:36871), 4faca80c18564946aa198a96659d1cf5 (127.20.178.67:37711)
I20260929 03:17:38.334065 22605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52d1395edcc64b50a8e4546a461dffc3" candidate_uuid: "64ec142b9e1f41cc808a076c2fcf4409" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fb0d1e69eba4b7998056ad25fcbc335"
I20260929 03:17:38.334147 22605 raft_consensus.cc:3037] T 52d1395edcc64b50a8e4546a461dffc3 P 5fb0d1e69eba4b7998056ad25fcbc335 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:17:38.334167 22745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52d1395edcc64b50a8e4546a461dffc3" candidate_uuid: "64ec142b9e1f41cc808a076c2fcf4409" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4faca80c18564946aa198a96659d1cf5"
I20260929 03:17:38.334268 22745 raft_consensus.cc:3037] T 52d1395edcc64b50a8e4546a461dffc3 P 4faca80c18564946aa198a96659d1cf5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:17:38.334791 22605 raft_consensus.cc:2454] T 52d1395edcc64b50a8e4546a461dffc3 P 5fb0d1e69eba4b7998056ad25fcbc335 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64ec142b9e1f41cc808a076c2fcf4409 in term 1.
I20260929 03:17:38.334959 22399 leader_election.cc:302] T 52d1395edcc64b50a8e4546a461dffc3 P 64ec142b9e1f41cc808a076c2fcf4409 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5fb0d1e69eba4b7998056ad25fcbc335, 64ec142b9e1f41cc808a076c2fcf4409; no voters:
I20260929 03:17:38.335235 22745 raft_consensus.cc:2454] T 52d1395edcc64b50a8e4546a461dffc3 P 4faca80c18564946aa198a96659d1cf5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64ec142b9e1f41cc808a076c2fcf4409 in term 1.
I20260929 03:17:38.335345 22821 raft_consensus.cc:2781] T 52d1395edcc64b50a8e4546a461dffc3 P 64ec142b9e1f41cc808a076c2fcf4409 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:17:38.335461 22821 raft_consensus.cc:686] T 52d1395edcc64b50a8e4546a461dffc3 P 64ec142b9e1f41cc808a076c2fcf4409 [term 1 LEADER]: Becoming Leader. State: Replica: 64ec142b9e1f41cc808a076c2fcf4409, State: Running, Role: LEADER
I20260929 03:17:38.335533 22821 consensus_queue.cc:237] T 52d1395edcc64b50a8e4546a461dffc3 P 64ec142b9e1f41cc808a076c2fcf4409 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ec142b9e1f41cc808a076c2fcf4409" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 32925 } } peers { permanent_uuid: "5fb0d1e69eba4b7998056ad25fcbc335" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 36871 } } peers { permanent_uuid: "4faca80c18564946aa198a96659d1cf5" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 37711 } }
I20260929 03:17:38.336289 22306 catalog_manager.cc:5495] T 52d1395edcc64b50a8e4546a461dffc3 P 64ec142b9e1f41cc808a076c2fcf4409 reported cstate change: term changed from 0 to 1, leader changed from <none> to 64ec142b9e1f41cc808a076c2fcf4409 (127.20.178.65). New cstate: current_term: 1 leader_uuid: "64ec142b9e1f41cc808a076c2fcf4409" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ec142b9e1f41cc808a076c2fcf4409" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 32925 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5fb0d1e69eba4b7998056ad25fcbc335" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 36871 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4faca80c18564946aa198a96659d1cf5" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 37711 } health_report { overall_health: UNKNOWN } } }
W20260929 03:17:38.345753 22792 tablet.cc:2369] T 52d1395edcc64b50a8e4546a461dffc3 P 4faca80c18564946aa198a96659d1cf5: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 29 03:17:38 dist-test-slave-c9qh krb5kdc[22263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651857, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.126@KRBTEST.COM for kudu/127.20.178.126@KRBTEST.COM
I20260929 03:17:38.368227 22851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:38.361341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:36403 (local address 127.0.0.1:32942)
0929 03:17:38.361516 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0929 03:17:38.361525 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 03:17:38.361584 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:17:38.361728 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:17:38.361729 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:17:38.362004 (+ 275us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:17:38.362051 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:38.362053 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:38.362971 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:38.362972 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:38.363490 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:38.363491 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:38.363885 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:38.363885 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:38.363924 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:38.364450 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:17:38.364455 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:17:38.365832 (+ 1377us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:17:38.367317 (+ 1485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:17:38.367324 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:17:38.367325 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:17:38.367503 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:17:38.367669 (+ 166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:17:38.367670 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:17:38.367670 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:17:38.367725 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:17:38.367947 (+ 222us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:17:38.367953 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:17:38.367991 (+ 38us) client_negotiation.cc:769] Sending connection context
0929 03:17:38.368042 (+ 51us) client_negotiation.cc:240] Negotiation successful
0929 03:17:38.368125 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":93,"threads_started":1}
I20260929 03:17:38.368309 22358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:38.361429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32942 (local address 127.20.178.126:36403)
0929 03:17:38.361493 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:38.361496 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:38.361591 (+ 95us) server_negotiation.cc:407] Connection header received
0929 03:17:38.361626 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:38.361628 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:38.361639 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:38.361717 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:17:38.362117 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:38.362883 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:38.363570 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:38.363810 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:38.363881 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:38.365953 (+ 2072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:17:38.365954 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:17:38.365954 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:17:38.365961 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:17:38.367246 (+ 1285us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:38.367567 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:38.367567 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:38.367568 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:38.367614 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:38.367775 (+ 161us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:38.367776 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:38.367776 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:38.367879 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:17:38.367946 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0929 03:17:38.368076 (+ 130us) server_negotiation.cc:299] Negotiation successful
0929 03:17:38.368167 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":32,"server-negotiator.queue_time_us":37}
I20260929 03:17:38.370391 22306 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.20.178.126@KRBTEST.COM'} at 127.0.0.1:32942:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260929 03:17:38.370666 22306 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 03:17:38.372102 22445 tablet_service.cc:1467] Processing CreateTablet for tablet 5a512b4d0e7a44128eb4cd7b8e2b1292 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=4fe4d8bf8bb647ec9f97f666e5599a1d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 03:17:38.372103 22725 tablet_service.cc:1467] Processing CreateTablet for tablet 5a512b4d0e7a44128eb4cd7b8e2b1292 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=4fe4d8bf8bb647ec9f97f666e5599a1d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 03:17:38.372153 22585 tablet_service.cc:1467] Processing CreateTablet for tablet 5a512b4d0e7a44128eb4cd7b8e2b1292 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=4fe4d8bf8bb647ec9f97f666e5599a1d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 03:17:38.372254 22725 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a512b4d0e7a44128eb4cd7b8e2b1292. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:17:38.372262 22585 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a512b4d0e7a44128eb4cd7b8e2b1292. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:17:38.372282 22445 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a512b4d0e7a44128eb4cd7b8e2b1292. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:17:38.373792 22815 tablet_bootstrap.cc:492] T 5a512b4d0e7a44128eb4cd7b8e2b1292 P 64ec142b9e1f41cc808a076c2fcf4409: Bootstrap starting.
I20260929 03:17:38.373804 22814 tablet_bootstrap.cc:492] T 5a512b4d0e7a44128eb4cd7b8e2b1292 P 4faca80c18564946aa198a96659d1cf5: Bootstrap starting.
I20260929 03:17:38.373840 22818 tablet_bootstrap.cc:492] T 5a512b4d0e7a44128eb4cd7b8e2b1292 P 5fb0d1e69eba4b7998056ad25fcbc335: Bootstrap starting.
I20260929 03:17:38.374127 22814 tablet_bootstrap.cc:654] T 5a512b4d0e7a44128eb4cd7b8e2b1292 P 4faca80c18564946aa198a96659d1cf5: Neither blocks nor log segments found. Creating new log.
I20260929 03:17:38.374256 22815 tablet_bootstrap.cc:654] T 5a512b4d0e7a44128eb4cd7b8e2b1292 P 64ec142b9e1f41cc808a076c2fcf4409: Neither blocks nor log segments found. Creating new log.
I20260929 03:17:38.374262 22818 tablet_bootstrap.cc:654] T 5a512b4d0e7a44128eb4cd7b8e2b1292 P 5fb0d1e69eba4b7998056ad25fcbc335: Neither blocks nor log segments found. Creating new log.
I20260929 03:17:38.374655 22814 tablet_bootstrap.cc:492] T 5a512b4d0e7a44128eb4cd7b8e2b1292 P 4faca80c18564946aa198a96659d1cf5: No bootstrap required, opened a new log
I20260929 03:17:38.374722 22814 ts_tablet_manager.cc:1397] T 5a512b4d0e7a44128eb4cd7b8e2b1292 P 4faca80c18564946aa198a96659d1cf5: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260929 03:17:38.374826 22814 raft_consensus.cc:348] T 5a512b4d0e7a44128eb4cd7b8e2b1292 P 4faca80c18564946aa198a96659d1cf5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ec142b9e1f41cc808a076c2fcf4409" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 32925 } } peers { permanent_uuid: "5fb0d1e69eba4b7998056ad25fcbc335" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 36871 } } peers { permanent_uuid: "4faca80c18564946aa198a96659d1cf5" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 37711 } }
I20260929 03:17:38.374877 22814 raft_consensus.cc:374] T 5a512b4d0e7a44128eb4cd7b8e2b1292 P 4faca80c18564946aa198a96659d1cf5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:17:38.374907 22814 raft_consensus.cc:729] T 5a512b4d0e7a44128eb4cd7b8e2b1292 P 4faca80c18564946aa198a96659d1cf5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4faca80c18564946aa198a96659d1cf5, State: Initialized, Role: FOLLOWER
I20260929 03:17:38.374872 22818 tablet_bootstrap.cc:492] T 5a512b4d0e7a44128eb4cd7b8e2b1292 P 5fb0d1e69eba4b7998056ad25fcbc335: No bootstrap required, opened a new log
I20260929 03:17:38.374966 22818 ts_tablet_manager.cc:1397] T 5a512b4d0e7a44128eb4cd7b8e2b1292 P 5fb0d1e69eba4b7998056ad25fcbc335: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260929 03:17:38.374960 22814 consensus_queue.cc:260] T 5a512b4d0e7a44128eb4cd7b8e2b1292 P 4faca80c18564946aa198a96659d1cf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ec142b9e1f41cc808a076c2fcf4409" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 32925 } } peers { permanent_uuid: "5fb0d1e69eba4b7998056ad25fcbc335" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 36871 } } peers { permanent_uuid: "4faca80c18564946aa198a96659d1cf5" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 37711 } }
I20260929 03:17:38.375047 22814 ts_tablet_manager.cc:1428] T 5a512b4d0e7a44128eb4cd7b8e2b1292 P 4faca80c18564946aa198a96659d1cf5: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260929 03:17:38.375063 22824 tablet_replica.cc:439] T 5a512b4d0e7a44128eb4cd7b8e2b1292 P 4faca80c18564946aa198a96659d1cf5: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ec142b9e1f41cc808a076c2fcf4409" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 32925 } } peers { permanent_uuid: "5fb0d1e69eba4b7998056ad25fcbc335" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 36871 } } peers { permanent_uuid: "4faca80c18564946aa198a96659d1cf5" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 37711 } } }
I20260929 03:17:38.375136 22824 tablet_replica.cc:442] T 5a512b4d0e7a44128eb4cd7b8e2b1292 P 4faca80c18564946aa198a96659d1cf5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 03:17:38.375113 22818 raft_consensus.cc:348] T 5a512b4d0e7a44128eb4cd7b8e2b1292 P 5fb0d1e69eba4b7998056ad25fcbc335 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ec142b9e1f41cc808a076c2fcf4409" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 32925 } } peers { permanent_uuid: "5fb0d1e69eba4b7998056ad25fcbc335" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 36871 } } peers { permanent_uuid: "4faca80c18564946aa198a96659d1cf5" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 37711 } }
I20260929 03:17:38.375177 22818 raft_consensus.cc:374] T 5a512b4d0e7a44128eb4cd7b8e2b1292 P 5fb0d1e69eba4b7998056ad25fcbc335 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:17:38.375203 22818 raft_consensus.cc:729] T 5a512b4d0e7a44128eb4cd7b8e2b1292 P 5fb0d1e69eba4b7998056ad25fcbc335 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fb0d1e69eba4b7998056ad25fcbc335, State: Initialized, Role: FOLLOWER
I20260929 03:17:38.375262 22818 consensus_queue.cc:260] T 5a512b4d0e7a44128eb4cd7b8e2b1292 P 5fb0d1e69eba4b7998056ad25fcbc335 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ec142b9e1f41cc808a076c2fcf4409" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 32925 } } peers { permanent_uuid: "5fb0d1e69eba4b7998056ad25fcbc335" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 36871 } } peers { permanent_uuid: "4faca80c18564946aa198a96659d1cf5" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 37711 } }
I20260929 03:17:38.375360 22818 ts_tablet_manager.cc:1428] T 5a512b4d0e7a44128eb4cd7b8e2b1292 P 5fb0d1e69eba4b7998056ad25fcbc335: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260929 03:17:38.375463 22815 tablet_bootstrap.cc:492] T 5a512b4d0e7a44128eb4cd7b8e2b1292 P 64ec142b9e1f41cc808a076c2fcf4409: No bootstrap required, opened a new log
I20260929 03:17:38.375516 22815 ts_tablet_manager.cc:1397] T 5a512b4d0e7a44128eb4cd7b8e2b1292 P 64ec142b9e1f41cc808a076c2fcf4409: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 03:17:38.375490 22822 tablet_replica.cc:439] T 5a512b4d0e7a44128eb4cd7b8e2b1292 P 5fb0d1e69eba4b7998056ad25fcbc335: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ec142b9e1f41cc808a076c2fcf4409" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 32925 } } peers { permanent_uuid: "5fb0d1e69eba4b7998056ad25fcbc335" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 36871 } } peers { permanent_uuid: "4faca80c18564946aa198a96659d1cf5" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 37711 } } }
I20260929 03:17:38.375581 22822 tablet_replica.cc:442] T 5a512b4d0e7a44128eb4cd7b8e2b1292 P 5fb0d1e69eba4b7998056ad25fcbc335: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 03:17:38.375659 22815 raft_consensus.cc:348] T 5a512b4d0e7a44128eb4cd7b8e2b1292 P 64ec142b9e1f41cc808a076c2fcf4409 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ec142b9e1f41cc808a076c2fcf4409" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 32925 } } peers { permanent_uuid: "5fb0d1e69eba4b7998056ad25fcbc335" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 36871 } } peers { permanent_uuid: "4faca80c18564946aa198a96659d1cf5" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 37711 } }
I20260929 03:17:38.375746 22815 raft_consensus.cc:374] T 5a512b4d0e7a44128eb4cd7b8e2b1292 P 64ec142b9e1f41cc808a076c2fcf4409 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:17:38.375772 22815 raft_consensus.cc:729] T 5a512b4d0e7a44128eb4cd7b8e2b1292 P 64ec142b9e1f41cc808a076c2fcf4409 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64ec142b9e1f41cc808a076c2fcf4409, State: Initialized, Role: FOLLOWER
I20260929 03:17:38.375811 22815 consensus_queue.cc:260] T 5a512b4d0e7a44128eb4cd7b8e2b1292 P 64ec142b9e1f41cc808a076c2fcf4409 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ec142b9e1f41cc808a076c2fcf4409" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 32925 } } peers { permanent_uuid: "5fb0d1e69eba4b7998056ad25fcbc335" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 36871 } } peers { permanent_uuid: "4faca80c18564946aa198a96659d1cf5" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 37711 } }
I20260929 03:17:38.375897 22815 ts_tablet_manager.cc:1428] T 5a512b4d0e7a44128eb4cd7b8e2b1292 P 64ec142b9e1f41cc808a076c2fcf4409: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260929 03:17:38.375998 22844 tablet_replica.cc:439] T 5a512b4d0e7a44128eb4cd7b8e2b1292 P 64ec142b9e1f41cc808a076c2fcf4409: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ec142b9e1f41cc808a076c2fcf4409" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 32925 } } peers { permanent_uuid: "5fb0d1e69eba4b7998056ad25fcbc335" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 36871 } } peers { permanent_uuid: "4faca80c18564946aa198a96659d1cf5" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 37711 } } }
I20260929 03:17:38.376103 22844 tablet_replica.cc:442] T 5a512b4d0e7a44128eb4cd7b8e2b1292 P 64ec142b9e1f41cc808a076c2fcf4409: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 03:17:38.510893 22822 raft_consensus.cc:482] T 5a512b4d0e7a44128eb4cd7b8e2b1292 P 5fb0d1e69eba4b7998056ad25fcbc335 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:17:38.511011 22822 raft_consensus.cc:504] T 5a512b4d0e7a44128eb4cd7b8e2b1292 P 5fb0d1e69eba4b7998056ad25fcbc335 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ec142b9e1f41cc808a076c2fcf4409" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 32925 } } peers { permanent_uuid: "5fb0d1e69eba4b7998056ad25fcbc335" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 36871 } } peers { permanent_uuid: "4faca80c18564946aa198a96659d1cf5" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 37711 } }
I20260929 03:17:38.511435 22822 leader_election.cc:288] T 5a512b4d0e7a44128eb4cd7b8e2b1292 P 5fb0d1e69eba4b7998056ad25fcbc335 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 64ec142b9e1f41cc808a076c2fcf4409 (127.20.178.65:32925), 4faca80c18564946aa198a96659d1cf5 (127.20.178.67:37711)
I20260929 03:17:38.514585 22813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:38.512101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.66:47105 (local address 127.20.178.67:37711)
0929 03:17:38.512145 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:38.512147 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:38.512156 (+ 9us) server_negotiation.cc:407] Connection header received
0929 03:17:38.512241 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:38.512242 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:38.512252 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:38.512305 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:17:38.512465 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:38.513029 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:38.513894 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:38.514240 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:38.514296 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:38.514302 (+ 6us) server_negotiation.cc:1035] Waiting for connection context
0929 03:17:38.514472 (+ 170us) server_negotiation.cc:299] Negotiation successful
0929 03:17:38.514517 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260929 03:17:38.514599 22856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:38.511986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.67:37711 (local address 127.20.178.66:47105)
0929 03:17:38.512108 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0929 03:17:38.512119 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 03:17:38.512169 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:17:38.512318 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:17:38.512319 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:17:38.512321 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:17:38.512381 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:38.512382 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:38.513113 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:38.513114 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:38.513804 (+ 690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:38.513805 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:38.514322 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:38.514323 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:38.514394 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:38.514398 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 03:17:38.514451 (+ 53us) client_negotiation.cc:240] Negotiation successful
0929 03:17:38.514480 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":62,"threads_started":1}
I20260929 03:17:38.514645 22810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:38.511717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.66:42819 (local address 127.20.178.65:32925)
0929 03:17:38.511809 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:38.511811 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:38.511967 (+ 156us) server_negotiation.cc:407] Connection header received
0929 03:17:38.512028 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:38.512029 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:38.512037 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:38.512129 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:17:38.512327 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:38.512903 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:38.513908 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:38.514290 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:38.514360 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:38.514368 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0929 03:17:38.514531 (+ 163us) server_negotiation.cc:299] Negotiation successful
0929 03:17:38.514570 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 03:17:38.514909 22855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:38.511629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.65:32925 (local address 127.20.178.66:42819)
0929 03:17:38.511867 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0929 03:17:38.511880 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 03:17:38.511957 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:17:38.512134 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:17:38.512134 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:17:38.512137 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:17:38.512238 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:38.512240 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:38.512981 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:38.512982 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:38.513798 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:38.513800 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:38.514372 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:38.514373 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:38.514429 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:38.514449 (+ 20us) client_negotiation.cc:769] Sending connection context
0929 03:17:38.514506 (+ 57us) client_negotiation.cc:240] Negotiation successful
0929 03:17:38.514544 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":218,"mutex_wait_us":11,"thread_start_us":192,"threads_started":1}
I20260929 03:17:38.514933 22745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a512b4d0e7a44128eb4cd7b8e2b1292" candidate_uuid: "5fb0d1e69eba4b7998056ad25fcbc335" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4faca80c18564946aa198a96659d1cf5" is_pre_election: true
I20260929 03:17:38.515019 22745 raft_consensus.cc:2454] T 5a512b4d0e7a44128eb4cd7b8e2b1292 P 4faca80c18564946aa198a96659d1cf5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5fb0d1e69eba4b7998056ad25fcbc335 in term 0.
I20260929 03:17:38.515168 22536 leader_election.cc:302] T 5a512b4d0e7a44128eb4cd7b8e2b1292 P 5fb0d1e69eba4b7998056ad25fcbc335 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4faca80c18564946aa198a96659d1cf5, 5fb0d1e69eba4b7998056ad25fcbc335; no voters:
I20260929 03:17:38.515319 22822 raft_consensus.cc:2781] T 5a512b4d0e7a44128eb4cd7b8e2b1292 P 5fb0d1e69eba4b7998056ad25fcbc335 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 03:17:38.515367 22822 raft_consensus.cc:482] T 5a512b4d0e7a44128eb4cd7b8e2b1292 P 5fb0d1e69eba4b7998056ad25fcbc335 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 03:17:38.515388 22822 raft_consensus.cc:3037] T 5a512b4d0e7a44128eb4cd7b8e2b1292 P 5fb0d1e69eba4b7998056ad25fcbc335 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:17:38.515383 22465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a512b4d0e7a44128eb4cd7b8e2b1292" candidate_uuid: "5fb0d1e69eba4b7998056ad25fcbc335" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64ec142b9e1f41cc808a076c2fcf4409" is_pre_election: true
I20260929 03:17:38.515457 22465 raft_consensus.cc:2454] T 5a512b4d0e7a44128eb4cd7b8e2b1292 P 64ec142b9e1f41cc808a076c2fcf4409 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5fb0d1e69eba4b7998056ad25fcbc335 in term 0.
I20260929 03:17:38.516041 22822 raft_consensus.cc:504] T 5a512b4d0e7a44128eb4cd7b8e2b1292 P 5fb0d1e69eba4b7998056ad25fcbc335 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ec142b9e1f41cc808a076c2fcf4409" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 32925 } } peers { permanent_uuid: "5fb0d1e69eba4b7998056ad25fcbc335" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 36871 } } peers { permanent_uuid: "4faca80c18564946aa198a96659d1cf5" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 37711 } }
I20260929 03:17:38.516134 22822 leader_election.cc:288] T 5a512b4d0e7a44128eb4cd7b8e2b1292 P 5fb0d1e69eba4b7998056ad25fcbc335 [CANDIDATE]: Term 1 election: Requested vote from peers 64ec142b9e1f41cc808a076c2fcf4409 (127.20.178.65:32925), 4faca80c18564946aa198a96659d1cf5 (127.20.178.67:37711)
I20260929 03:17:38.516312 22745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a512b4d0e7a44128eb4cd7b8e2b1292" candidate_uuid: "5fb0d1e69eba4b7998056ad25fcbc335" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4faca80c18564946aa198a96659d1cf5"
I20260929 03:17:38.516331 22465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a512b4d0e7a44128eb4cd7b8e2b1292" candidate_uuid: "5fb0d1e69eba4b7998056ad25fcbc335" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64ec142b9e1f41cc808a076c2fcf4409"
I20260929 03:17:38.516392 22745 raft_consensus.cc:3037] T 5a512b4d0e7a44128eb4cd7b8e2b1292 P 4faca80c18564946aa198a96659d1cf5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:17:38.516408 22465 raft_consensus.cc:3037] T 5a512b4d0e7a44128eb4cd7b8e2b1292 P 64ec142b9e1f41cc808a076c2fcf4409 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:17:38.517062 22745 raft_consensus.cc:2454] T 5a512b4d0e7a44128eb4cd7b8e2b1292 P 4faca80c18564946aa198a96659d1cf5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5fb0d1e69eba4b7998056ad25fcbc335 in term 1.
I20260929 03:17:38.517072 22465 raft_consensus.cc:2454] T 5a512b4d0e7a44128eb4cd7b8e2b1292 P 64ec142b9e1f41cc808a076c2fcf4409 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5fb0d1e69eba4b7998056ad25fcbc335 in term 1.
I20260929 03:17:38.517220 22536 leader_election.cc:302] T 5a512b4d0e7a44128eb4cd7b8e2b1292 P 5fb0d1e69eba4b7998056ad25fcbc335 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5fb0d1e69eba4b7998056ad25fcbc335, 64ec142b9e1f41cc808a076c2fcf4409; no voters:
I20260929 03:17:38.517323 22822 raft_consensus.cc:2781] T 5a512b4d0e7a44128eb4cd7b8e2b1292 P 5fb0d1e69eba4b7998056ad25fcbc335 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:17:38.517431 22822 raft_consensus.cc:686] T 5a512b4d0e7a44128eb4cd7b8e2b1292 P 5fb0d1e69eba4b7998056ad25fcbc335 [term 1 LEADER]: Becoming Leader. State: Replica: 5fb0d1e69eba4b7998056ad25fcbc335, State: Running, Role: LEADER
I20260929 03:17:38.517494 22822 consensus_queue.cc:237] T 5a512b4d0e7a44128eb4cd7b8e2b1292 P 5fb0d1e69eba4b7998056ad25fcbc335 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ec142b9e1f41cc808a076c2fcf4409" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 32925 } } peers { permanent_uuid: "5fb0d1e69eba4b7998056ad25fcbc335" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 36871 } } peers { permanent_uuid: "4faca80c18564946aa198a96659d1cf5" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 37711 } }
I20260929 03:17:38.517807 22857 tablet_replica.cc:439] T 5a512b4d0e7a44128eb4cd7b8e2b1292 P 5fb0d1e69eba4b7998056ad25fcbc335: TxnStatusTablet state changed. Reason: New leader 5fb0d1e69eba4b7998056ad25fcbc335. Latest consensus state: current_term: 1 leader_uuid: "5fb0d1e69eba4b7998056ad25fcbc335" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ec142b9e1f41cc808a076c2fcf4409" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 32925 } } peers { permanent_uuid: "5fb0d1e69eba4b7998056ad25fcbc335" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 36871 } } peers { permanent_uuid: "4faca80c18564946aa198a96659d1cf5" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 37711 } } }
I20260929 03:17:38.517908 22857 tablet_replica.cc:442] T 5a512b4d0e7a44128eb4cd7b8e2b1292 P 5fb0d1e69eba4b7998056ad25fcbc335: This TxnStatusTablet replica's current role is: LEADER
I20260929 03:17:38.518079 22859 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260929 03:17:38.518190 22859 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260929 03:17:38.518164 22308 catalog_manager.cc:5495] T 5a512b4d0e7a44128eb4cd7b8e2b1292 P 5fb0d1e69eba4b7998056ad25fcbc335 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5fb0d1e69eba4b7998056ad25fcbc335 (127.20.178.66). New cstate: current_term: 1 leader_uuid: "5fb0d1e69eba4b7998056ad25fcbc335" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ec142b9e1f41cc808a076c2fcf4409" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 32925 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5fb0d1e69eba4b7998056ad25fcbc335" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 36871 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4faca80c18564946aa198a96659d1cf5" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 37711 } health_report { overall_health: UNKNOWN } } }
I20260929 03:17:38.558413 22812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:38.555747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47058 (local address 127.20.178.66:36871)
0929 03:17:38.555790 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:38.555792 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:38.555803 (+ 11us) server_negotiation.cc:407] Connection header received
0929 03:17:38.555890 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:38.555891 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:38.555907 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:38.555966 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:17:38.556099 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:38.556575 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:38.557689 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:38.557868 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:38.557917 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:38.558104 (+ 187us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:17:38.558192 (+ 88us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:17:38.558244 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0929 03:17:38.558328 (+ 84us) server_negotiation.cc:299] Negotiation successful
0929 03:17:38.558349 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260929 03:17:38.558461 22851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:38.555672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.66:36871 (local address 127.0.0.1:47058)
0929 03:17:38.555756 (+ 84us) negotiation.cc:107] Waiting for socket to connect
0929 03:17:38.555767 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 03:17:38.555814 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:17:38.555988 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:17:38.555989 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:17:38.555991 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 03:17:38.556044 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:38.556047 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:38.556671 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:38.556672 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:38.557592 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:38.557595 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:38.557934 (+ 339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:38.557935 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:38.558003 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:38.558019 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 03:17:38.558251 (+ 232us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 03:17:38.558276 (+ 25us) client_negotiation.cc:769] Sending connection context
0929 03:17:38.558326 (+ 50us) client_negotiation.cc:240] Negotiation successful
0929 03:17:38.558377 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260929 03:17:38.559620 22465 raft_consensus.cc:1261] T 5a512b4d0e7a44128eb4cd7b8e2b1292 P 64ec142b9e1f41cc808a076c2fcf4409 [term 1 FOLLOWER]: Refusing update from remote peer 5fb0d1e69eba4b7998056ad25fcbc335: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:17:38.559748 22745 raft_consensus.cc:1261] T 5a512b4d0e7a44128eb4cd7b8e2b1292 P 4faca80c18564946aa198a96659d1cf5 [term 1 FOLLOWER]: Refusing update from remote peer 5fb0d1e69eba4b7998056ad25fcbc335: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:17:38.560073 22822 consensus_queue.cc:1035] T 5a512b4d0e7a44128eb4cd7b8e2b1292 P 5fb0d1e69eba4b7998056ad25fcbc335 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64ec142b9e1f41cc808a076c2fcf4409" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 32925 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:17:38.560184 22857 consensus_queue.cc:1035] T 5a512b4d0e7a44128eb4cd7b8e2b1292 P 5fb0d1e69eba4b7998056ad25fcbc335 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4faca80c18564946aa198a96659d1cf5" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 37711 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:17:38.560755 22844 tablet_replica.cc:439] T 5a512b4d0e7a44128eb4cd7b8e2b1292 P 64ec142b9e1f41cc808a076c2fcf4409: TxnStatusTablet state changed. Reason: New leader 5fb0d1e69eba4b7998056ad25fcbc335. Latest consensus state: current_term: 1 leader_uuid: "5fb0d1e69eba4b7998056ad25fcbc335" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ec142b9e1f41cc808a076c2fcf4409" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 32925 } } peers { permanent_uuid: "5fb0d1e69eba4b7998056ad25fcbc335" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 36871 } } peers { permanent_uuid: "4faca80c18564946aa198a96659d1cf5" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 37711 } } }
I20260929 03:17:38.560865 22844 tablet_replica.cc:442] T 5a512b4d0e7a44128eb4cd7b8e2b1292 P 64ec142b9e1f41cc808a076c2fcf4409: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 03:17:38.561306 22824 tablet_replica.cc:439] T 5a512b4d0e7a44128eb4cd7b8e2b1292 P 4faca80c18564946aa198a96659d1cf5: TxnStatusTablet state changed. Reason: New leader 5fb0d1e69eba4b7998056ad25fcbc335. Latest consensus state: current_term: 1 leader_uuid: "5fb0d1e69eba4b7998056ad25fcbc335" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ec142b9e1f41cc808a076c2fcf4409" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 32925 } } peers { permanent_uuid: "5fb0d1e69eba4b7998056ad25fcbc335" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 36871 } } peers { permanent_uuid: "4faca80c18564946aa198a96659d1cf5" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 37711 } } }
I20260929 03:17:38.561396 22824 tablet_replica.cc:442] T 5a512b4d0e7a44128eb4cd7b8e2b1292 P 4faca80c18564946aa198a96659d1cf5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 03:17:38.561903 22857 tablet_replica.cc:439] T 5a512b4d0e7a44128eb4cd7b8e2b1292 P 5fb0d1e69eba4b7998056ad25fcbc335: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5fb0d1e69eba4b7998056ad25fcbc335" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ec142b9e1f41cc808a076c2fcf4409" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 32925 } } peers { permanent_uuid: "5fb0d1e69eba4b7998056ad25fcbc335" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 36871 } } peers { permanent_uuid: "4faca80c18564946aa198a96659d1cf5" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 37711 } } }
I20260929 03:17:38.561985 22857 tablet_replica.cc:442] T 5a512b4d0e7a44128eb4cd7b8e2b1292 P 5fb0d1e69eba4b7998056ad25fcbc335: This TxnStatusTablet replica's current role is: LEADER
I20260929 03:17:38.561928 22864 tablet_replica.cc:439] T 5a512b4d0e7a44128eb4cd7b8e2b1292 P 5fb0d1e69eba4b7998056ad25fcbc335: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5fb0d1e69eba4b7998056ad25fcbc335" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ec142b9e1f41cc808a076c2fcf4409" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 32925 } } peers { permanent_uuid: "5fb0d1e69eba4b7998056ad25fcbc335" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 36871 } } peers { permanent_uuid: "4faca80c18564946aa198a96659d1cf5" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 37711 } } }
I20260929 03:17:38.562019 22864 tablet_replica.cc:442] T 5a512b4d0e7a44128eb4cd7b8e2b1292 P 5fb0d1e69eba4b7998056ad25fcbc335: This TxnStatusTablet replica's current role is: LEADER
I20260929 03:17:38.562085 22844 tablet_replica.cc:439] T 5a512b4d0e7a44128eb4cd7b8e2b1292 P 64ec142b9e1f41cc808a076c2fcf4409: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5fb0d1e69eba4b7998056ad25fcbc335" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ec142b9e1f41cc808a076c2fcf4409" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 32925 } } peers { permanent_uuid: "5fb0d1e69eba4b7998056ad25fcbc335" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 36871 } } peers { permanent_uuid: "4faca80c18564946aa198a96659d1cf5" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 37711 } } }
I20260929 03:17:38.562148 22844 tablet_replica.cc:442] T 5a512b4d0e7a44128eb4cd7b8e2b1292 P 64ec142b9e1f41cc808a076c2fcf4409: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 03:17:38.562908 22828 mvcc.cc:205] Tried to move back new op lower bound from 7334510012658360320 to 7334510012488224768. Current Snapshot: MvccSnapshot[applied={T|T < 7334510012658360320 or (T in {7334510012658360320})}]
I20260929 03:17:38.563107 22824 tablet_replica.cc:439] T 5a512b4d0e7a44128eb4cd7b8e2b1292 P 4faca80c18564946aa198a96659d1cf5: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5fb0d1e69eba4b7998056ad25fcbc335" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ec142b9e1f41cc808a076c2fcf4409" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 32925 } } peers { permanent_uuid: "5fb0d1e69eba4b7998056ad25fcbc335" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 36871 } } peers { permanent_uuid: "4faca80c18564946aa198a96659d1cf5" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 37711 } } }
I20260929 03:17:38.563186 22824 tablet_replica.cc:442] T 5a512b4d0e7a44128eb4cd7b8e2b1292 P 4faca80c18564946aa198a96659d1cf5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 03:17:38.566749 22810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:38.564158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52796 (local address 127.20.178.65:32925)
0929 03:17:38.564197 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:38.564199 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:38.564207 (+ 8us) server_negotiation.cc:407] Connection header received
0929 03:17:38.564276 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:38.564277 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:38.564296 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:38.564374 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:17:38.564503 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:38.565274 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:38.565957 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:38.566257 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:38.566329 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:38.566447 (+ 118us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:17:38.566561 (+ 114us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:17:38.566629 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0929 03:17:38.566663 (+ 34us) server_negotiation.cc:299] Negotiation successful
0929 03:17:38.566680 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260929 03:17:38.571583 22812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:38.569068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.65:49045 (local address 127.20.178.66:36871)
0929 03:17:38.569115 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:38.569116 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:38.569255 (+ 139us) server_negotiation.cc:407] Connection header received
0929 03:17:38.569297 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:38.569297 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:38.569305 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:38.569353 (+ 48us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:17:38.569475 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:38.569897 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:38.570977 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:38.571145 (+ 168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:38.571189 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:38.571389 (+ 200us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:17:38.571412 (+ 23us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:17:38.571449 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0929 03:17:38.571522 (+ 73us) server_negotiation.cc:299] Negotiation successful
0929 03:17:38.571537 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260929 03:17:38.571653 22867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:38.569014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.66:36871 (local address 127.20.178.65:49045)
0929 03:17:38.569174 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0929 03:17:38.569185 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 03:17:38.569246 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:17:38.569367 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:17:38.569368 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:17:38.569369 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 03:17:38.569421 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:38.569422 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:38.569960 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:38.569961 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:38.570894 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:38.570896 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:38.571218 (+ 322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:38.571219 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:38.571279 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:38.571293 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 03:17:38.571467 (+ 174us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 03:17:38.571468 (+ 1us) client_negotiation.cc:769] Sending connection context
0929 03:17:38.571522 (+ 54us) client_negotiation.cc:240] Negotiation successful
0929 03:17:38.571561 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":91,"threads_started":1}
I20260929 03:17:38.574020 22605 raft_consensus.cc:1261] T 52d1395edcc64b50a8e4546a461dffc3 P 5fb0d1e69eba4b7998056ad25fcbc335 [term 1 FOLLOWER]: Refusing update from remote peer 64ec142b9e1f41cc808a076c2fcf4409: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:17:38.574048 22745 raft_consensus.cc:1261] T 52d1395edcc64b50a8e4546a461dffc3 P 4faca80c18564946aa198a96659d1cf5 [term 1 FOLLOWER]: Refusing update from remote peer 64ec142b9e1f41cc808a076c2fcf4409: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:17:38.574307 22821 consensus_queue.cc:1035] T 52d1395edcc64b50a8e4546a461dffc3 P 64ec142b9e1f41cc808a076c2fcf4409 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5fb0d1e69eba4b7998056ad25fcbc335" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 36871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:17:38.574476 22821 consensus_queue.cc:1035] T 52d1395edcc64b50a8e4546a461dffc3 P 64ec142b9e1f41cc808a076c2fcf4409 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4faca80c18564946aa198a96659d1cf5" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 37711 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:17:38.584362 22810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:38.581546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.66:52661 (local address 127.20.178.65:32925)
0929 03:17:38.581582 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:38.581584 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:38.581833 (+ 249us) server_negotiation.cc:407] Connection header received
0929 03:17:38.581922 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:38.581923 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:38.581932 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:38.581970 (+ 38us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:17:38.582072 (+ 102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:38.582604 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:38.583522 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:38.583852 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:38.583947 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:38.584129 (+ 182us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:17:38.584161 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:17:38.584209 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0929 03:17:38.584275 (+ 66us) server_negotiation.cc:299] Negotiation successful
0929 03:17:38.584293 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260929 03:17:38.584394 22871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:38.581514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.65:32925 (local address 127.20.178.66:52661)
0929 03:17:38.581765 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0929 03:17:38.581777 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 03:17:38.581864 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:17:38.581981 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:17:38.581982 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:17:38.581983 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 03:17:38.582027 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:38.582028 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:38.582683 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:38.582684 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:38.583448 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:38.583450 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:38.583947 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:38.583948 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:38.584012 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:38.584027 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 03:17:38.584216 (+ 189us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 03:17:38.584218 (+ 2us) client_negotiation.cc:769] Sending connection context
0929 03:17:38.584272 (+ 54us) client_negotiation.cc:240] Negotiation successful
0929 03:17:38.584315 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":75,"threads_started":1}
I20260929 03:17:38.594094 22308 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37942:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:17:38.594205 22308 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37942:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:17:38.594903 22308 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 721722c2f2ff42e788510074c1a2d3fd: Sending DeleteTablet for 3 replicas of tablet 52d1395edcc64b50a8e4546a461dffc3
I20260929 03:17:38.595206 22445 tablet_service.cc:1514] Processing DeleteTablet for tablet 52d1395edcc64b50a8e4546a461dffc3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:17:38 UTC) from {username='kudu', principal='kudu/127.20.178.126@KRBTEST.COM'} at 127.0.0.1:33346
I20260929 03:17:38.595280 22585 tablet_service.cc:1514] Processing DeleteTablet for tablet 52d1395edcc64b50a8e4546a461dffc3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:17:38 UTC) from {username='kudu', principal='kudu/127.20.178.126@KRBTEST.COM'} at 127.0.0.1:39168
I20260929 03:17:38.595305 22725 tablet_service.cc:1514] Processing DeleteTablet for tablet 52d1395edcc64b50a8e4546a461dffc3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:17:38 UTC) from {username='kudu', principal='kudu/127.20.178.126@KRBTEST.COM'} at 127.0.0.1:59232
I20260929 03:17:38.595403 22838 tablet_replica.cc:330] T 52d1395edcc64b50a8e4546a461dffc3 P 64ec142b9e1f41cc808a076c2fcf4409: stopping tablet replica
I20260929 03:17:38.595451 22838 raft_consensus.cc:2229] T 52d1395edcc64b50a8e4546a461dffc3 P 64ec142b9e1f41cc808a076c2fcf4409 [term 1 LEADER]: Raft consensus shutting down.
I20260929 03:17:38.595532 22838 raft_consensus.cc:2258] T 52d1395edcc64b50a8e4546a461dffc3 P 64ec142b9e1f41cc808a076c2fcf4409 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:17:38.595602 22837 tablet_replica.cc:330] T 52d1395edcc64b50a8e4546a461dffc3 P 5fb0d1e69eba4b7998056ad25fcbc335: stopping tablet replica
I20260929 03:17:38.595665 22837 raft_consensus.cc:2229] T 52d1395edcc64b50a8e4546a461dffc3 P 5fb0d1e69eba4b7998056ad25fcbc335 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:17:38.595698 22837 raft_consensus.cc:2258] T 52d1395edcc64b50a8e4546a461dffc3 P 5fb0d1e69eba4b7998056ad25fcbc335 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:17:38.595815 22836 tablet_replica.cc:330] T 52d1395edcc64b50a8e4546a461dffc3 P 4faca80c18564946aa198a96659d1cf5: stopping tablet replica
I20260929 03:17:38.595889 22836 raft_consensus.cc:2229] T 52d1395edcc64b50a8e4546a461dffc3 P 4faca80c18564946aa198a96659d1cf5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:17:38.595925 22836 raft_consensus.cc:2258] T 52d1395edcc64b50a8e4546a461dffc3 P 4faca80c18564946aa198a96659d1cf5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:17:38.596524 22837 ts_tablet_manager.cc:1905] T 52d1395edcc64b50a8e4546a461dffc3 P 5fb0d1e69eba4b7998056ad25fcbc335: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:17:38.598268 22837 ts_tablet_manager.cc:1918] T 52d1395edcc64b50a8e4546a461dffc3 P 5fb0d1e69eba4b7998056ad25fcbc335: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 03:17:38.598320 22837 log.cc:1196] T 52d1395edcc64b50a8e4546a461dffc3 P 5fb0d1e69eba4b7998056ad25fcbc335: Deleting WAL directory at /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651852476208-21193-0/minicluster-data/ts-1/wal/wals/52d1395edcc64b50a8e4546a461dffc3
I20260929 03:17:38.598531 22837 ts_tablet_manager.cc:1939] T 52d1395edcc64b50a8e4546a461dffc3 P 5fb0d1e69eba4b7998056ad25fcbc335: Deleting consensus metadata
I20260929 03:17:38.598786 22836 ts_tablet_manager.cc:1905] T 52d1395edcc64b50a8e4546a461dffc3 P 4faca80c18564946aa198a96659d1cf5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:17:38.598896 22295 catalog_manager.cc:4841] TS 5fb0d1e69eba4b7998056ad25fcbc335 (127.20.178.66:36871): tablet 52d1395edcc64b50a8e4546a461dffc3 (table test-table [id=af9fd6429e7b4dc7aea0a7196588bc35]) successfully deleted
I20260929 03:17:38.600319 22836 ts_tablet_manager.cc:1918] T 52d1395edcc64b50a8e4546a461dffc3 P 4faca80c18564946aa198a96659d1cf5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 03:17:38.600369 22836 log.cc:1196] T 52d1395edcc64b50a8e4546a461dffc3 P 4faca80c18564946aa198a96659d1cf5: Deleting WAL directory at /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651852476208-21193-0/minicluster-data/ts-2/wal/wals/52d1395edcc64b50a8e4546a461dffc3
I20260929 03:17:38.600574 22836 ts_tablet_manager.cc:1939] T 52d1395edcc64b50a8e4546a461dffc3 P 4faca80c18564946aa198a96659d1cf5: Deleting consensus metadata
I20260929 03:17:38.600906 22292 catalog_manager.cc:4841] TS 4faca80c18564946aa198a96659d1cf5 (127.20.178.67:37711): tablet 52d1395edcc64b50a8e4546a461dffc3 (table test-table [id=af9fd6429e7b4dc7aea0a7196588bc35]) successfully deleted
I20260929 03:17:38.601028 22838 ts_tablet_manager.cc:1905] T 52d1395edcc64b50a8e4546a461dffc3 P 64ec142b9e1f41cc808a076c2fcf4409: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:17:38.602613 22838 ts_tablet_manager.cc:1918] T 52d1395edcc64b50a8e4546a461dffc3 P 64ec142b9e1f41cc808a076c2fcf4409: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 03:17:38.602674 22838 log.cc:1196] T 52d1395edcc64b50a8e4546a461dffc3 P 64ec142b9e1f41cc808a076c2fcf4409: Deleting WAL directory at /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651852476208-21193-0/minicluster-data/ts-0/wal/wals/52d1395edcc64b50a8e4546a461dffc3
I20260929 03:17:38.602890 22838 ts_tablet_manager.cc:1939] T 52d1395edcc64b50a8e4546a461dffc3 P 64ec142b9e1f41cc808a076c2fcf4409: Deleting consensus metadata
I20260929 03:17:38.603273 22292 catalog_manager.cc:4841] TS 64ec142b9e1f41cc808a076c2fcf4409 (127.20.178.65:32925): tablet 52d1395edcc64b50a8e4546a461dffc3 (table test-table [id=af9fd6429e7b4dc7aea0a7196588bc35]) successfully deleted
Sep 29 03:17:38 dist-test-slave-c9qh krb5kdc[22263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651858, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.20.178.65@KRBTEST.COM
I20260929 03:17:38.614462 22810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:38.604072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52804 (local address 127.20.178.65:32925)
0929 03:17:38.604141 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:38.604143 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:38.604382 (+ 239us) server_negotiation.cc:407] Connection header received
0929 03:17:38.604433 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:38.604434 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:38.604445 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:38.604511 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:17:38.605205 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:38.605927 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:38.607004 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:38.607290 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:38.607370 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:38.610961 (+ 3591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:17:38.610968 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:17:38.610969 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:17:38.610977 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:17:38.613121 (+ 2144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:38.613534 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:38.613537 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:38.613538 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:38.613616 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:38.613838 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:38.613839 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:38.613840 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:38.614016 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:17:38.614164 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0929 03:17:38.614255 (+ 91us) server_negotiation.cc:299] Negotiation successful
0929 03:17:38.614336 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
W20260929 03:17:38.614701 22505 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52804
I20260929 03:17:38.623201 22358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:38.615653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32952 (local address 127.20.178.126:36403)
0929 03:17:38.615699 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:38.615701 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:38.615936 (+ 235us) server_negotiation.cc:407] Connection header received
0929 03:17:38.615986 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:38.615988 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:38.616000 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:38.616060 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:17:38.616682 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:38.617433 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:38.618453 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:38.618761 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:38.618839 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:38.620196 (+ 1357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:17:38.620197 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:17:38.620198 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:17:38.620205 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:17:38.621915 (+ 1710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:38.622332 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:38.622333 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:38.622334 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:38.622396 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:38.622630 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:38.622638 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:38.622638 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:38.622775 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:17:38.622873 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0929 03:17:38.623025 (+ 152us) server_negotiation.cc:299] Negotiation successful
0929 03:17:38.623107 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
W20260929 03:17:38.623644 22308 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:32952
I20260929 03:17:38.624336 21193 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuoaHbX/build/release/bin/kudu with pid 22380
I20260929 03:17:38.629325 21193 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuoaHbX/build/release/bin/kudu with pid 22520
I20260929 03:17:38.634474 21193 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuoaHbX/build/release/bin/kudu with pid 22660
I20260929 03:17:38.639662 21193 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuoaHbX/build/release/bin/kudu with pid 22276
2026-09-29T03:17:38Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2315 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651852476208-21193-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:17:38 dist-test-slave-c9qh krb5kdc[22881](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:17:38 dist-test-slave-c9qh krb5kdc[22881](info): set up 2 sockets
Sep 29 03:17:38 dist-test-slave-c9qh krb5kdc[22881](info): commencing operation
krb5kdc: starting...
W20260929 03:17:39.668294 21193 mini_kdc.cc:121] Time spent starting KDC: real 1.008s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651852476208-21193-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651852476208-21193-0/krb5kdc/test-user.keytab.
Sep 29 03:17:39 dist-test-slave-c9qh krb5kdc[22881](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651859, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:17:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:17:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.178.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.178.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.178.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.126.keytab.
Entry for principal kudu/127.20.178.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.126.keytab.
I20260929 03:17:39.779793 21193 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuoaHbX/build/release/bin/kudu
/tmp/dist-test-taskuoaHbX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651852476208-21193-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651852476208-21193-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651852476208-21193-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651852476208-21193-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.178.126:40307
--webserver_interface=127.20.178.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.126.keytab
--principal=kudu/127.20.178.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.178.84:43491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.178.126:40307
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651852476208-21193-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651852476208-21193-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651852476208-21193-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:17:39.862080 22894 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:17:39.862326 22894 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:17:39.862365 22894 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:17:39.863541 22894 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:17:39.863616 22894 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260929 03:17:39.863639 22894 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:17:39.863662 22894 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:17:39.863693 22894 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:17:39.863718 22894 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:17:39.865285 22894 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.178.84:43491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651852476208-21193-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651852476208-21193-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.178.126:40307
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.126.keytab
--principal=kudu/127.20.178.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.178.126:40307
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651852476208-21193-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.178.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22894
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651852476208-21193-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:17:39.865577 22894 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:17:39.865814 22894 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 03:17:39.870821 22894 server_base.cc:1034] running on GCE node
W20260929 03:17:39.870776 22900 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:17:39.870841 22902 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:17:39.870805 22899 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:17:39.871260 22894 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:17:39.871546 22894 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:17:39.872705 22894 hybrid_clock.cc:648] HybridClock initialized: now 1790651859872684 us; error 41 us; skew 500 ppm
Sep 29 03:17:39 dist-test-slave-c9qh krb5kdc[22881](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651859, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:17:39.875190 22894 init.cc:377] Logged in from keytab as kudu/127.20.178.126@KRBTEST.COM (short username kudu)
I20260929 03:17:39.875602 22894 webserver.cc:460] Webserver started at http://127.20.178.126:40087/ using document root <none> and password file <none>
I20260929 03:17:39.875828 22894 fs_manager.cc:359] Metadata directory not provided
I20260929 03:17:39.875895 22894 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:17:39.876045 22894 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:17:39.877174 22894 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651852476208-21193-0/minicluster-data/master-0/data/instance:
uuid: "21c6f00f395f4ca0af3dc2144bf92b8f"
format_stamp: "Formatted at 2026-09-29 03:17:39 on dist-test-slave-c9qh"
server_key: "a53f7cd01f2520da0658b6f3c71a022b"
server_key_iv: "6ed946e35b47bb341d2efede255b1eba"
server_key_version: "encryptionkey@0"
I20260929 03:17:39.877529 22894 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651852476208-21193-0/minicluster-data/master-0/wal/instance:
uuid: "21c6f00f395f4ca0af3dc2144bf92b8f"
format_stamp: "Formatted at 2026-09-29 03:17:39 on dist-test-slave-c9qh"
server_key: "a53f7cd01f2520da0658b6f3c71a022b"
server_key_iv: "6ed946e35b47bb341d2efede255b1eba"
server_key_version: "encryptionkey@0"
I20260929 03:17:39.878839 22894 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 03:17:39.879539 22909 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:17:39.879715 22894 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 03:17:39.879823 22894 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651852476208-21193-0/minicluster-data/master-0/data,/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651852476208-21193-0/minicluster-data/master-0/wal
uuid: "21c6f00f395f4ca0af3dc2144bf92b8f"
format_stamp: "Formatted at 2026-09-29 03:17:39 on dist-test-slave-c9qh"
server_key: "a53f7cd01f2520da0658b6f3c71a022b"
server_key_iv: "6ed946e35b47bb341d2efede255b1eba"
server_key_version: "encryptionkey@0"
I20260929 03:17:39.879900 22894 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651852476208-21193-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651852476208-21193-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651852476208-21193-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:17:39.894259 22894 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:17:39.896027 22894 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:17:39.896196 22894 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:17:39.901119 22894 rpc_server.cc:307] RPC server started. Bound to: 127.20.178.126:40307
I20260929 03:17:39.901204 22971 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.178.126:40307 every 8 connection(s)
I20260929 03:17:39.901531 22894 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651852476208-21193-0/minicluster-data/master-0/data/info.pb
I20260929 03:17:39.902179 22972 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:17:39.904378 21193 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuoaHbX/build/release/bin/kudu as pid 22894
I20260929 03:17:39.904418 22972 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 21c6f00f395f4ca0af3dc2144bf92b8f: Bootstrap starting.
I20260929 03:17:39.904484 21193 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651852476208-21193-0/minicluster-data/master-0/wal/instance
I20260929 03:17:39.904726 21193 external_mini_cluster.cc:1428] Setting key 8f1556fa350f0af02c729cd9ed302801
I20260929 03:17:39.905175 22972 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 21c6f00f395f4ca0af3dc2144bf92b8f: Neither blocks nor log segments found. Creating new log.
I20260929 03:17:39.905514 22972 log.cc:824] T 00000000000000000000000000000000 P 21c6f00f395f4ca0af3dc2144bf92b8f: Log is configured to *not* fsync() on all Append() calls
I20260929 03:17:39.906267 22972 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 21c6f00f395f4ca0af3dc2144bf92b8f: No bootstrap required, opened a new log
I20260929 03:17:39.908564 22972 raft_consensus.cc:348] T 00000000000000000000000000000000 P 21c6f00f395f4ca0af3dc2144bf92b8f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21c6f00f395f4ca0af3dc2144bf92b8f" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 40307 } }
I20260929 03:17:39.908759 22972 raft_consensus.cc:374] T 00000000000000000000000000000000 P 21c6f00f395f4ca0af3dc2144bf92b8f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:17:39.908787 22972 raft_consensus.cc:729] T 00000000000000000000000000000000 P 21c6f00f395f4ca0af3dc2144bf92b8f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21c6f00f395f4ca0af3dc2144bf92b8f, State: Initialized, Role: FOLLOWER
I20260929 03:17:39.908933 22972 consensus_queue.cc:260] T 00000000000000000000000000000000 P 21c6f00f395f4ca0af3dc2144bf92b8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21c6f00f395f4ca0af3dc2144bf92b8f" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 40307 } }
I20260929 03:17:39.909003 22972 raft_consensus.cc:388] T 00000000000000000000000000000000 P 21c6f00f395f4ca0af3dc2144bf92b8f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:17:39.909024 22972 raft_consensus.cc:482] T 00000000000000000000000000000000 P 21c6f00f395f4ca0af3dc2144bf92b8f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:17:39.909101 22972 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 21c6f00f395f4ca0af3dc2144bf92b8f [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:17:39.909739 22972 raft_consensus.cc:504] T 00000000000000000000000000000000 P 21c6f00f395f4ca0af3dc2144bf92b8f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21c6f00f395f4ca0af3dc2144bf92b8f" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 40307 } }
I20260929 03:17:39.909847 22972 leader_election.cc:302] T 00000000000000000000000000000000 P 21c6f00f395f4ca0af3dc2144bf92b8f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 21c6f00f395f4ca0af3dc2144bf92b8f; no voters:
I20260929 03:17:39.910009 22972 leader_election.cc:288] T 00000000000000000000000000000000 P 21c6f00f395f4ca0af3dc2144bf92b8f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:17:39.910058 22977 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 21c6f00f395f4ca0af3dc2144bf92b8f [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:17:39.910272 22977 raft_consensus.cc:686] T 00000000000000000000000000000000 P 21c6f00f395f4ca0af3dc2144bf92b8f [term 1 LEADER]: Becoming Leader. State: Replica: 21c6f00f395f4ca0af3dc2144bf92b8f, State: Running, Role: LEADER
I20260929 03:17:39.910393 22972 sys_catalog.cc:564] T 00000000000000000000000000000000 P 21c6f00f395f4ca0af3dc2144bf92b8f [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:17:39.910379 22977 consensus_queue.cc:237] T 00000000000000000000000000000000 P 21c6f00f395f4ca0af3dc2144bf92b8f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21c6f00f395f4ca0af3dc2144bf92b8f" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 40307 } }
I20260929 03:17:39.910794 22979 sys_catalog.cc:455] T 00000000000000000000000000000000 P 21c6f00f395f4ca0af3dc2144bf92b8f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 21c6f00f395f4ca0af3dc2144bf92b8f. Latest consensus state: current_term: 1 leader_uuid: "21c6f00f395f4ca0af3dc2144bf92b8f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21c6f00f395f4ca0af3dc2144bf92b8f" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 40307 } } }
I20260929 03:17:39.910887 22979 sys_catalog.cc:458] T 00000000000000000000000000000000 P 21c6f00f395f4ca0af3dc2144bf92b8f [sys.catalog]: This master's current role is: LEADER
I20260929 03:17:39.911113 22983 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:17:39.911634 22983 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:17:39.912065 22978 sys_catalog.cc:455] T 00000000000000000000000000000000 P 21c6f00f395f4ca0af3dc2144bf92b8f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "21c6f00f395f4ca0af3dc2144bf92b8f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21c6f00f395f4ca0af3dc2144bf92b8f" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 40307 } } }
I20260929 03:17:39.912150 22978 sys_catalog.cc:458] T 00000000000000000000000000000000 P 21c6f00f395f4ca0af3dc2144bf92b8f [sys.catalog]: This master's current role is: LEADER
Sep 29 03:17:39 dist-test-slave-c9qh krb5kdc[22881](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651859, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.178.126@KRBTEST.COM
I20260929 03:17:39.913606 22983 catalog_manager.cc:1347] Generated new cluster ID: 243bafbb222b4af1b0734c2486212041
I20260929 03:17:39.913662 22983 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:17:39.916615 22975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:39.905214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37152 (local address 127.20.178.126:40307)
0929 03:17:39.905387 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:39.905396 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:39.905429 (+ 33us) server_negotiation.cc:407] Connection header received
0929 03:17:39.906018 (+ 589us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:39.906027 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:39.906089 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:39.906237 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:17:39.906842 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:39.908274 (+ 1432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:39.909245 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:39.909874 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:39.909992 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:39.912992 (+ 3000us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:17:39.912995 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:17:39.912996 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:17:39.913008 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:17:39.914995 (+ 1987us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:39.915410 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:39.915413 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:39.915413 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:39.915511 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:39.915727 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:39.915728 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:39.915728 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:39.916004 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:17:39.916147 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0929 03:17:39.916288 (+ 141us) server_negotiation.cc:299] Negotiation successful
0929 03:17:39.916412 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":83,"threads_started":1}
I20260929 03:17:39.927848 22983 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:17:39.928347 22983 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:17:39.938627 22983 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 21c6f00f395f4ca0af3dc2144bf92b8f: Generated new TSK 0
I20260929 03:17:39.938843 22983 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.178.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.178.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.178.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.65.keytab.
Entry for principal kudu/127.20.178.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.65.keytab.
I20260929 03:17:39.967051 21193 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuoaHbX/build/release/bin/kudu
/tmp/dist-test-taskuoaHbX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651852476208-21193-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651852476208-21193-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651852476208-21193-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651852476208-21193-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.178.65:0
--local_ip_for_outbound_sockets=127.20.178.65
--webserver_interface=127.20.178.65
--webserver_port=0
--tserver_master_addrs=127.20.178.126:40307
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.65.keytab
--principal=kudu/127.20.178.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.178.84:43491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651852476208-21193-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651852476208-21193-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651852476208-21193-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:17:40.047094 22998 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:17:40.047299 22998 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:17:40.047322 22998 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:17:40.048460 22998 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:17:40.048503 22998 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:17:40.048523 22998 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260929 03:17:40.048559 22998 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 03:17:40.048595 22998 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.178.65
I20260929 03:17:40.050199 22998 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.178.84:43491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651852476208-21193-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651852476208-21193-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.65.keytab
--principal=kudu/127.20.178.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.178.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651852476208-21193-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.178.65
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.20.178.126:40307
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22998
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.178.65
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651852476208-21193-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:17:40.050434 22998 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:17:40.050679 22998 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:17:40.055734 23004 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:17:40.055755 23003 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:17:40.056015 22998 server_base.cc:1034] running on GCE node
W20260929 03:17:40.055734 23006 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:17:40.056275 22998 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:17:40.056529 22998 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:17:40.057693 22998 hybrid_clock.cc:648] HybridClock initialized: now 1790651860057684 us; error 47 us; skew 500 ppm
Sep 29 03:17:40 dist-test-slave-c9qh krb5kdc[22881](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651860, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:17:40.060267 22998 init.cc:377] Logged in from keytab as kudu/127.20.178.65@KRBTEST.COM (short username kudu)
I20260929 03:17:40.060731 22998 webserver.cc:460] Webserver started at http://127.20.178.65:35185/ using document root <none> and password file <none>
I20260929 03:17:40.060963 22998 fs_manager.cc:359] Metadata directory not provided
I20260929 03:17:40.061029 22998 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:17:40.061182 22998 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:17:40.062207 22998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651852476208-21193-0/minicluster-data/ts-0/data/instance:
uuid: "7502a5d6be1646cba535265e54b2f07e"
format_stamp: "Formatted at 2026-09-29 03:17:40 on dist-test-slave-c9qh"
server_key: "b33d01fb1e43451575bc5459b521bff6"
server_key_iv: "9172aff85952e660fc6c8fb19ff4bb1c"
server_key_version: "encryptionkey@0"
I20260929 03:17:40.062564 22998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651852476208-21193-0/minicluster-data/ts-0/wal/instance:
uuid: "7502a5d6be1646cba535265e54b2f07e"
format_stamp: "Formatted at 2026-09-29 03:17:40 on dist-test-slave-c9qh"
server_key: "b33d01fb1e43451575bc5459b521bff6"
server_key_iv: "9172aff85952e660fc6c8fb19ff4bb1c"
server_key_version: "encryptionkey@0"
I20260929 03:17:40.063885 22998 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 03:17:40.064730 23013 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:17:40.064921 22998 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 03:17:40.065012 22998 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651852476208-21193-0/minicluster-data/ts-0/data,/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651852476208-21193-0/minicluster-data/ts-0/wal
uuid: "7502a5d6be1646cba535265e54b2f07e"
format_stamp: "Formatted at 2026-09-29 03:17:40 on dist-test-slave-c9qh"
server_key: "b33d01fb1e43451575bc5459b521bff6"
server_key_iv: "9172aff85952e660fc6c8fb19ff4bb1c"
server_key_version: "encryptionkey@0"
I20260929 03:17:40.065100 22998 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651852476208-21193-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651852476208-21193-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651852476208-21193-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:17:40.086320 22998 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:17:40.088303 22998 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:17:40.088472 22998 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:17:40.089057 22998 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:17:40.089125 22998 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:17:40.089166 22998 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:17:40.089195 22998 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 03:17:40 dist-test-slave-c9qh krb5kdc[22881](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651860, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.65@KRBTEST.COM for kudu/127.20.178.126@KRBTEST.COM
I20260929 03:17:40.096725 22998 rpc_server.cc:307] RPC server started. Bound to: 127.20.178.65:43333
I20260929 03:17:40.096799 23128 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.178.65:43333 every 8 connection(s)
I20260929 03:17:40.097124 22998 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651852476208-21193-0/minicluster-data/ts-0/data/info.pb
I20260929 03:17:40.099511 22975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:40.089651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.65:51357 (local address 127.20.178.126:40307)
0929 03:17:40.089730 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:40.089733 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:40.089964 (+ 231us) server_negotiation.cc:407] Connection header received
0929 03:17:40.090743 (+ 779us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:40.090745 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:40.090758 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:40.090867 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:17:40.091588 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:40.092532 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:40.093663 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:40.093909 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:40.094006 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:40.096881 (+ 2875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:17:40.096883 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:17:40.096883 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:17:40.096892 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:17:40.098358 (+ 1466us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:40.098755 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:40.098756 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:40.098756 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:40.098803 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:40.099011 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:40.099011 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:40.099011 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:40.099105 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:17:40.099161 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0929 03:17:40.099378 (+ 217us) server_negotiation.cc:299] Negotiation successful
0929 03:17:40.099447 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 03:17:40.099735 23022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:40.089545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:40307 (local address 127.20.178.65:51357)
0929 03:17:40.089815 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0929 03:17:40.089863 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0929 03:17:40.090580 (+ 717us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:17:40.090968 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:17:40.090972 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:17:40.091230 (+ 258us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:17:40.091438 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:40.091443 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:40.092665 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:40.092667 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:40.093545 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:40.093547 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:40.094045 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:40.094047 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:40.094167 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:40.094745 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:17:40.094755 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:17:40.096794 (+ 2039us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:17:40.098449 (+ 1655us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:17:40.098455 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:17:40.098456 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:17:40.098686 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:17:40.098858 (+ 172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:17:40.098858 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:17:40.098858 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:17:40.098955 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:17:40.099169 (+ 214us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:17:40.099174 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:17:40.099281 (+ 107us) client_negotiation.cc:769] Sending connection context
0929 03:17:40.099358 (+ 77us) client_negotiation.cc:240] Negotiation successful
0929 03:17:40.099479 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":108,"threads_started":1}
I20260929 03:17:40.100265 23129 heartbeater.cc:347] Connected to a master server at 127.20.178.126:40307
I20260929 03:17:40.100342 23129 heartbeater.cc:464] Registering TS with master...
I20260929 03:17:40.100517 23129 heartbeater.cc:511] Master 127.20.178.126:40307 requested a full tablet report, sending...
I20260929 03:17:40.101045 22925 ts_manager.cc:194] Registered new tserver with Master: 7502a5d6be1646cba535265e54b2f07e (127.20.178.65:43333)
I20260929 03:17:40.101801 22925 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.178.65@KRBTEST.COM'} at 127.20.178.65:51357
I20260929 03:17:40.101972 21193 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuoaHbX/build/release/bin/kudu as pid 22998
I20260929 03:17:40.102087 21193 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651852476208-21193-0/minicluster-data/ts-0/wal/instance
I20260929 03:17:40.102283 21193 external_mini_cluster.cc:1428] Setting key 99172bd134696f3f5f967e739f0b95dc
I20260929 03:17:40.107206 22975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:40.101506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.65:44467 (local address 127.20.178.126:40307)
0929 03:17:40.101543 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:40.101545 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:40.101664 (+ 119us) server_negotiation.cc:407] Connection header received
0929 03:17:40.101681 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:40.101682 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:40.101690 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:40.101741 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:17:40.102100 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:40.102697 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:40.103494 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:40.103675 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:40.103721 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:40.104676 (+ 955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:17:40.104678 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:17:40.104678 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:17:40.104684 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:17:40.106067 (+ 1383us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:40.106455 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:40.106456 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:40.106456 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:40.106508 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:40.106712 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:40.106713 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:40.106713 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:40.106819 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:17:40.106890 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0929 03:17:40.107053 (+ 163us) server_negotiation.cc:299] Negotiation successful
0929 03:17:40.107123 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260929 03:17:40.107213 23135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:40.101448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:40307 (local address 127.20.178.65:44467)
0929 03:17:40.101565 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0929 03:17:40.101575 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 03:17:40.101630 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:17:40.101752 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:17:40.101753 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:17:40.101964 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:17:40.102030 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:40.102032 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:40.102766 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:40.102766 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:40.103410 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:40.103413 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:40.103741 (+ 328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:40.103742 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:40.103825 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:40.104181 (+ 356us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:17:40.104187 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:17:40.104562 (+ 375us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:17:40.106167 (+ 1605us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:17:40.106168 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:17:40.106168 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:17:40.106369 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:17:40.106588 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:17:40.106589 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:17:40.106589 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:17:40.106650 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:17:40.106895 (+ 245us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:17:40.106896 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:17:40.106938 (+ 42us) client_negotiation.cc:769] Sending connection context
0929 03:17:40.107023 (+ 85us) client_negotiation.cc:240] Negotiation successful
0929 03:17:40.107120 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":84,"thread_start_us":61,"threads_started":1}
WARNING: no policy specified for kudu/127.20.178.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.178.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.178.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.66.keytab.
Entry for principal kudu/127.20.178.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.66.keytab.
I20260929 03:17:40.129525 21193 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuoaHbX/build/release/bin/kudu
/tmp/dist-test-taskuoaHbX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651852476208-21193-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651852476208-21193-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651852476208-21193-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651852476208-21193-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.178.66:0
--local_ip_for_outbound_sockets=127.20.178.66
--webserver_interface=127.20.178.66
--webserver_port=0
--tserver_master_addrs=127.20.178.126:40307
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.66.keytab
--principal=kudu/127.20.178.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.178.84:43491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651852476208-21193-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651852476208-21193-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651852476208-21193-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:17:40.208737 23138 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:17:40.208930 23138 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:17:40.208953 23138 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:17:40.210103 23138 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:17:40.210144 23138 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:17:40.210165 23138 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260929 03:17:40.210176 23138 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 03:17:40.210238 23138 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.178.66
I20260929 03:17:40.211820 23138 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.178.84:43491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651852476208-21193-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651852476208-21193-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.66.keytab
--principal=kudu/127.20.178.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.178.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651852476208-21193-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.178.66
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.20.178.126:40307
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23138
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.178.66
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651852476208-21193-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:17:40.212061 23138 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:17:40.212324 23138 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:17:40.217123 23144 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:17:40.217151 23143 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:17:40.217135 23146 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:17:40.217409 23138 server_base.cc:1034] running on GCE node
I20260929 03:17:40.217576 23138 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:17:40.217777 23138 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:17:40.218914 23138 hybrid_clock.cc:648] HybridClock initialized: now 1790651860218904 us; error 27 us; skew 500 ppm
Sep 29 03:17:40 dist-test-slave-c9qh krb5kdc[22881](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651860, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:17:40.221325 23138 init.cc:377] Logged in from keytab as kudu/127.20.178.66@KRBTEST.COM (short username kudu)
I20260929 03:17:40.221735 23138 webserver.cc:460] Webserver started at http://127.20.178.66:45013/ using document root <none> and password file <none>
I20260929 03:17:40.221974 23138 fs_manager.cc:359] Metadata directory not provided
I20260929 03:17:40.222018 23138 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:17:40.222178 23138 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:17:40.223191 23138 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651852476208-21193-0/minicluster-data/ts-1/data/instance:
uuid: "d68a40cd55a24edbb306ac354251f160"
format_stamp: "Formatted at 2026-09-29 03:17:40 on dist-test-slave-c9qh"
server_key: "93774c724dd5c1742e24f9779b2b9378"
server_key_iv: "c277716b4a0c54c8e2ed74d054e16a70"
server_key_version: "encryptionkey@0"
I20260929 03:17:40.223592 23138 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651852476208-21193-0/minicluster-data/ts-1/wal/instance:
uuid: "d68a40cd55a24edbb306ac354251f160"
format_stamp: "Formatted at 2026-09-29 03:17:40 on dist-test-slave-c9qh"
server_key: "93774c724dd5c1742e24f9779b2b9378"
server_key_iv: "c277716b4a0c54c8e2ed74d054e16a70"
server_key_version: "encryptionkey@0"
I20260929 03:17:40.224932 23138 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 03:17:40.225653 23153 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:17:40.225842 23138 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 03:17:40.225942 23138 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651852476208-21193-0/minicluster-data/ts-1/data,/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651852476208-21193-0/minicluster-data/ts-1/wal
uuid: "d68a40cd55a24edbb306ac354251f160"
format_stamp: "Formatted at 2026-09-29 03:17:40 on dist-test-slave-c9qh"
server_key: "93774c724dd5c1742e24f9779b2b9378"
server_key_iv: "c277716b4a0c54c8e2ed74d054e16a70"
server_key_version: "encryptionkey@0"
I20260929 03:17:40.226009 23138 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651852476208-21193-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651852476208-21193-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651852476208-21193-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:17:40.256141 23138 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:17:40.258095 23138 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:17:40.258271 23138 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:17:40.258792 23138 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:17:40.258829 23138 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:17:40.258852 23138 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:17:40.258864 23138 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 03:17:40 dist-test-slave-c9qh krb5kdc[22881](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651860, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.66@KRBTEST.COM for kudu/127.20.178.126@KRBTEST.COM
I20260929 03:17:40.266112 23138 rpc_server.cc:307] RPC server started. Bound to: 127.20.178.66:34631
I20260929 03:17:40.266481 23138 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651852476208-21193-0/minicluster-data/ts-1/data/info.pb
I20260929 03:17:40.266453 23268 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.178.66:34631 every 8 connection(s)
I20260929 03:17:40.268927 22975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:40.259421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.66:40763 (local address 127.20.178.126:40307)
0929 03:17:40.259482 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:40.259484 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:40.259727 (+ 243us) server_negotiation.cc:407] Connection header received
0929 03:17:40.260263 (+ 536us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:40.260264 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:40.260276 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:40.260340 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:17:40.261365 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:40.261880 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:40.262904 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:40.263119 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:40.263211 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:40.265893 (+ 2682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:17:40.265895 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:17:40.265895 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:17:40.265903 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:17:40.267461 (+ 1558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:40.267919 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:40.267920 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:40.267921 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:40.267978 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:40.268252 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:40.268253 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:40.268253 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:40.268371 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:17:40.268443 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0929 03:17:40.268741 (+ 298us) server_negotiation.cc:299] Negotiation successful
0929 03:17:40.268818 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260929 03:17:40.269101 23162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:40.259412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:40307 (local address 127.20.178.66:40763)
0929 03:17:40.259599 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0929 03:17:40.259641 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 03:17:40.260165 (+ 524us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:17:40.260474 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:17:40.260479 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:17:40.260934 (+ 455us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:17:40.261244 (+ 310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:40.261253 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:40.261979 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:40.261980 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:40.262808 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:40.262810 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:40.263234 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:40.263235 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:40.263399 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:40.263899 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:17:40.263909 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:17:40.265795 (+ 1886us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:17:40.267554 (+ 1759us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:17:40.267561 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:17:40.267563 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:17:40.267839 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:17:40.268047 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:17:40.268048 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:17:40.268049 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:17:40.268167 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:17:40.268448 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:17:40.268454 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:17:40.268586 (+ 132us) client_negotiation.cc:769] Sending connection context
0929 03:17:40.268721 (+ 135us) client_negotiation.cc:240] Negotiation successful
0929 03:17:40.268877 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":105,"threads_started":1}
I20260929 03:17:40.269613 23269 heartbeater.cc:347] Connected to a master server at 127.20.178.126:40307
I20260929 03:17:40.269721 23269 heartbeater.cc:464] Registering TS with master...
I20260929 03:17:40.269904 23269 heartbeater.cc:511] Master 127.20.178.126:40307 requested a full tablet report, sending...
I20260929 03:17:40.270270 22926 ts_manager.cc:194] Registered new tserver with Master: d68a40cd55a24edbb306ac354251f160 (127.20.178.66:34631)
I20260929 03:17:40.270687 22926 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.178.66@KRBTEST.COM'} at 127.20.178.66:40763
I20260929 03:17:40.274286 21193 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuoaHbX/build/release/bin/kudu as pid 23138
I20260929 03:17:40.274389 21193 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651852476208-21193-0/minicluster-data/ts-1/wal/instance
I20260929 03:17:40.274585 21193 external_mini_cluster.cc:1428] Setting key b95d665867ffeb5e040ed35db101b952
I20260929 03:17:40.276364 23275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:40.270589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:40307 (local address 127.20.178.66:50643)
0929 03:17:40.270731 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0929 03:17:40.270747 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 03:17:40.270835 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:17:40.270980 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:17:40.270981 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:17:40.271144 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:17:40.271202 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:40.271203 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:40.271879 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:40.271879 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:40.272357 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:40.272358 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:40.272721 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:40.272722 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:40.272786 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:40.273200 (+ 414us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:17:40.273204 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:17:40.273460 (+ 256us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:17:40.274937 (+ 1477us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:17:40.274938 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:17:40.274938 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:17:40.275157 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:17:40.275482 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:17:40.275483 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:17:40.275484 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:17:40.275561 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:17:40.276012 (+ 451us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:17:40.276013 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:17:40.276075 (+ 62us) client_negotiation.cc:769] Sending connection context
0929 03:17:40.276140 (+ 65us) client_negotiation.cc:240] Negotiation successful
0929 03:17:40.276258 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"mutex_wait_us":5,"thread_start_us":61,"threads_started":1}
I20260929 03:17:40.276455 22975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:40.270651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.66:50643 (local address 127.20.178.126:40307)
0929 03:17:40.270691 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:40.270693 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:40.270812 (+ 119us) server_negotiation.cc:407] Connection header received
0929 03:17:40.270917 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:40.270918 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:40.270926 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:40.270985 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:17:40.271279 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:40.271795 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:40.272424 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:40.272601 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:40.272719 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:40.273526 (+ 807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:17:40.273526 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:17:40.273527 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:17:40.273531 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:17:40.274845 (+ 1314us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:40.275226 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:40.275228 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:40.275228 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:40.275298 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:40.275694 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:40.275695 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:40.275695 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:40.275938 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:17:40.276007 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0929 03:17:40.276205 (+ 198us) server_negotiation.cc:299] Negotiation successful
0929 03:17:40.276306 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
WARNING: no policy specified for kudu/127.20.178.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.178.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.178.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.67.keytab.
Entry for principal kudu/127.20.178.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.67.keytab.
I20260929 03:17:40.299983 21193 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuoaHbX/build/release/bin/kudu
/tmp/dist-test-taskuoaHbX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651852476208-21193-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651852476208-21193-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651852476208-21193-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651852476208-21193-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.178.67:0
--local_ip_for_outbound_sockets=127.20.178.67
--webserver_interface=127.20.178.67
--webserver_port=0
--tserver_master_addrs=127.20.178.126:40307
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.67.keytab
--principal=kudu/127.20.178.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.178.84:43491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651852476208-21193-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651852476208-21193-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651852476208-21193-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:17:40.380367 23278 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:17:40.380555 23278 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:17:40.380585 23278 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:17:40.381844 23278 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:17:40.381887 23278 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:17:40.381908 23278 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260929 03:17:40.381956 23278 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 03:17:40.381986 23278 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.178.67
I20260929 03:17:40.383558 23278 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.178.84:43491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651852476208-21193-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651852476208-21193-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.67.keytab
--principal=kudu/127.20.178.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.178.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651852476208-21193-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.178.67
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.20.178.126:40307
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23278
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.178.67
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651852476208-21193-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:17:40.383798 23278 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:17:40.384028 23278 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:17:40.388881 23286 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:17:40.388921 23283 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:17:40.388890 23284 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:17:40.389114 23278 server_base.cc:1034] running on GCE node
I20260929 03:17:40.389300 23278 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:17:40.389487 23278 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:17:40.390620 23278 hybrid_clock.cc:648] HybridClock initialized: now 1790651860390610 us; error 27 us; skew 500 ppm
Sep 29 03:17:40 dist-test-slave-c9qh krb5kdc[22881](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651860, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:17:40.393167 23278 init.cc:377] Logged in from keytab as kudu/127.20.178.67@KRBTEST.COM (short username kudu)
I20260929 03:17:40.393587 23278 webserver.cc:460] Webserver started at http://127.20.178.67:39467/ using document root <none> and password file <none>
I20260929 03:17:40.393831 23278 fs_manager.cc:359] Metadata directory not provided
I20260929 03:17:40.393899 23278 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:17:40.394044 23278 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:17:40.395095 23278 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651852476208-21193-0/minicluster-data/ts-2/data/instance:
uuid: "f87970e42bbb4b10884b1606288565ca"
format_stamp: "Formatted at 2026-09-29 03:17:40 on dist-test-slave-c9qh"
server_key: "be434f271e6306ca42d5fd3ac87a1a70"
server_key_iv: "930776b8f2d209a0e8c93d6bbb1f750d"
server_key_version: "encryptionkey@0"
I20260929 03:17:40.395447 23278 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651852476208-21193-0/minicluster-data/ts-2/wal/instance:
uuid: "f87970e42bbb4b10884b1606288565ca"
format_stamp: "Formatted at 2026-09-29 03:17:40 on dist-test-slave-c9qh"
server_key: "be434f271e6306ca42d5fd3ac87a1a70"
server_key_iv: "930776b8f2d209a0e8c93d6bbb1f750d"
server_key_version: "encryptionkey@0"
I20260929 03:17:40.396799 23278 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260929 03:17:40.397557 23293 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:17:40.397722 23278 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 03:17:40.397821 23278 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651852476208-21193-0/minicluster-data/ts-2/data,/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651852476208-21193-0/minicluster-data/ts-2/wal
uuid: "f87970e42bbb4b10884b1606288565ca"
format_stamp: "Formatted at 2026-09-29 03:17:40 on dist-test-slave-c9qh"
server_key: "be434f271e6306ca42d5fd3ac87a1a70"
server_key_iv: "930776b8f2d209a0e8c93d6bbb1f750d"
server_key_version: "encryptionkey@0"
I20260929 03:17:40.397897 23278 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651852476208-21193-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651852476208-21193-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651852476208-21193-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:17:40.412418 23278 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:17:40.414541 23278 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:17:40.414695 23278 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:17:40.415150 23278 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:17:40.415215 23278 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:17:40.415282 23278 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:17:40.415311 23278 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:17:40.421793 23278 rpc_server.cc:307] RPC server started. Bound to: 127.20.178.67:35231
I20260929 03:17:40.421852 23408 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.178.67:35231 every 8 connection(s)
Sep 29 03:17:40 dist-test-slave-c9qh krb5kdc[22881](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651860, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.67@KRBTEST.COM for kudu/127.20.178.126@KRBTEST.COM
I20260929 03:17:40.422154 23278 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651852476208-21193-0/minicluster-data/ts-2/data/info.pb
I20260929 03:17:40.425036 21193 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuoaHbX/build/release/bin/kudu as pid 23278
I20260929 03:17:40.425129 21193 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651852476208-21193-0/minicluster-data/ts-2/wal/instance
I20260929 03:17:40.425303 21193 external_mini_cluster.cc:1428] Setting key 9469650d34492ce068ffd710e250305a
I20260929 03:17:40.425343 22975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:40.415607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.67:59547 (local address 127.20.178.126:40307)
0929 03:17:40.415682 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:40.415684 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:40.415894 (+ 210us) server_negotiation.cc:407] Connection header received
0929 03:17:40.416324 (+ 430us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:40.416325 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:40.416338 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:40.416422 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:17:40.417183 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:40.418021 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:40.419564 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:40.419936 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:40.420016 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:40.422681 (+ 2665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:17:40.422682 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:17:40.422683 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:17:40.422691 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:17:40.424080 (+ 1389us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:40.424413 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:40.424414 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:40.424414 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:40.424462 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:40.424715 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:40.424715 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:40.424716 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:40.424825 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:17:40.424878 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0929 03:17:40.425142 (+ 264us) server_negotiation.cc:299] Negotiation successful
0929 03:17:40.425224 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 03:17:40.425426 23302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:40.415603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:40307 (local address 127.20.178.67:59547)
0929 03:17:40.415767 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0929 03:17:40.415806 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 03:17:40.416232 (+ 426us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:17:40.416484 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:17:40.416487 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:17:40.416797 (+ 310us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:17:40.417074 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:40.417081 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:40.418120 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:40.418121 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:40.419467 (+ 1346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:40.419470 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:40.420030 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:40.420031 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:40.420168 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:40.420655 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:17:40.420665 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:17:40.422584 (+ 1919us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:17:40.424153 (+ 1569us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:17:40.424157 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:17:40.424158 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:17:40.424352 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:17:40.424519 (+ 167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:17:40.424519 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:17:40.424519 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:17:40.424607 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:17:40.424894 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:17:40.424899 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:17:40.425002 (+ 103us) client_negotiation.cc:769] Sending connection context
0929 03:17:40.425074 (+ 72us) client_negotiation.cc:240] Negotiation successful
0929 03:17:40.425238 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":74,"threads_started":1}
I20260929 03:17:40.425824 23409 heartbeater.cc:347] Connected to a master server at 127.20.178.126:40307
I20260929 03:17:40.425956 23409 heartbeater.cc:464] Registering TS with master...
I20260929 03:17:40.426167 23409 heartbeater.cc:511] Master 127.20.178.126:40307 requested a full tablet report, sending...
I20260929 03:17:40.426517 22926 ts_manager.cc:194] Registered new tserver with Master: f87970e42bbb4b10884b1606288565ca (127.20.178.67:35231)
I20260929 03:17:40.426970 22926 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.178.67@KRBTEST.COM'} at 127.20.178.67:59547
I20260929 03:17:40.432112 23415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:40.426817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:40307 (local address 127.20.178.67:45729)
0929 03:17:40.426949 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0929 03:17:40.426960 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 03:17:40.427020 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:17:40.427156 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:17:40.427157 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:17:40.427373 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:17:40.427425 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:40.427427 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:40.428019 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:40.428019 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:40.428490 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:40.428491 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:40.428933 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:40.428934 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:40.429004 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:40.429422 (+ 418us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:17:40.429427 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:17:40.429693 (+ 266us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:17:40.431016 (+ 1323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:17:40.431017 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:17:40.431017 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:17:40.431189 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:17:40.431406 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:17:40.431407 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:17:40.431407 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:17:40.431463 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:17:40.431713 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:17:40.431714 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:17:40.431760 (+ 46us) client_negotiation.cc:769] Sending connection context
0929 03:17:40.431900 (+ 140us) client_negotiation.cc:240] Negotiation successful
0929 03:17:40.432017 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":83,"threads_started":1}
I20260929 03:17:40.432108 22975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:40.426900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.67:45729 (local address 127.20.178.126:40307)
0929 03:17:40.426941 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:40.426943 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:40.427010 (+ 67us) server_negotiation.cc:407] Connection header received
0929 03:17:40.427070 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:40.427070 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:40.427079 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:40.427138 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:17:40.427501 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:40.427932 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:40.428580 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:40.428846 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:40.428931 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:40.429821 (+ 890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:17:40.429823 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:17:40.429823 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:17:40.429829 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:17:40.430935 (+ 1106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:40.431283 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:40.431284 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:40.431285 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:40.431342 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:40.431535 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:40.431537 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:40.431537 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:40.431649 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:17:40.431722 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 03:17:40.431935 (+ 213us) server_negotiation.cc:299] Negotiation successful
0929 03:17:40.432014 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260929 03:17:40.436908 21193 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 03:17:40 dist-test-slave-c9qh krb5kdc[22881](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651860, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 03:17:40 dist-test-slave-c9qh krb5kdc[22881](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651860, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.178.126@KRBTEST.COM
I20260929 03:17:40.459084 22975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:40.452031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37156 (local address 127.20.178.126:40307)
0929 03:17:40.452094 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:40.452096 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:40.452104 (+ 8us) server_negotiation.cc:407] Connection header received
0929 03:17:40.452134 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:40.452135 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:40.452146 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:40.452230 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:17:40.452888 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:40.453377 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:40.454104 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:40.454295 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:40.454367 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:40.456537 (+ 2170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:17:40.456539 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:17:40.456539 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:17:40.456548 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:17:40.457863 (+ 1315us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:40.458277 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:40.458278 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:40.458279 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:40.458342 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:40.458561 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:40.458561 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:40.458562 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:40.458683 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:17:40.458797 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0929 03:17:40.458903 (+ 106us) server_negotiation.cc:299] Negotiation successful
0929 03:17:40.458988 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260929 03:17:40.460705 22926 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37156:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 03:17:40.462385 22926 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 03:17:40.469080 23426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:40.464663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.66:34631 (local address 127.0.0.1:38732)
0929 03:17:40.465369 (+ 706us) negotiation.cc:107] Waiting for socket to connect
0929 03:17:40.465379 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 03:17:40.465433 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:17:40.465647 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:17:40.465647 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:17:40.465649 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:17:40.465691 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:40.465692 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:40.466869 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:40.466870 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:40.467779 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:40.467781 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:40.468736 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:40.468737 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:40.468818 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:40.468835 (+ 17us) client_negotiation.cc:769] Sending connection context
0929 03:17:40.468909 (+ 74us) client_negotiation.cc:240] Negotiation successful
0929 03:17:40.468963 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":693,"mutex_wait_us":1,"thread_start_us":104,"threads_started":1}
I20260929 03:17:40.469099 23431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:40.464888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38732 (local address 127.20.178.66:34631)
0929 03:17:40.465059 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:40.465064 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:40.465511 (+ 447us) server_negotiation.cc:407] Connection header received
0929 03:17:40.465554 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:40.465556 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:40.465571 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:40.465642 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:17:40.465773 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:40.466787 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:40.467893 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:40.468612 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:40.468721 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:40.468789 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0929 03:17:40.468940 (+ 151us) server_negotiation.cc:299] Negotiation successful
0929 03:17:40.468989 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":75,"threads_started":1}
I20260929 03:17:40.469190 23428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:40.464623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.67:35231 (local address 127.0.0.1:49072)
0929 03:17:40.464826 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0929 03:17:40.464867 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 03:17:40.464930 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:17:40.465252 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:17:40.465255 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:17:40.465256 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:17:40.465322 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:40.465325 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:40.466358 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:40.466359 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:40.467754 (+ 1395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:40.467756 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:40.468826 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:40.468827 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:40.468915 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:40.468920 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 03:17:40.468973 (+ 53us) client_negotiation.cc:240] Negotiation successful
0929 03:17:40.469012 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":72,"threads_started":1}
I20260929 03:17:40.469358 23429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:40.464726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33500 (local address 127.20.178.65:43333)
0929 03:17:40.465093 (+ 367us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:40.465097 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:40.465108 (+ 11us) server_negotiation.cc:407] Connection header received
0929 03:17:40.465173 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:40.465175 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:40.465188 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:40.465287 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:17:40.465439 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:40.466567 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:40.468100 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:40.468779 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:40.468846 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:40.468899 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0929 03:17:40.469234 (+ 335us) server_negotiation.cc:299] Negotiation successful
0929 03:17:40.469275 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":80,"threads_started":1}
I20260929 03:17:40.469197 23427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:40.464654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.65:43333 (local address 127.0.0.1:33500)
0929 03:17:40.465054 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0929 03:17:40.465064 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 03:17:40.465101 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:17:40.465285 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:17:40.465286 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:17:40.465288 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:17:40.465348 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:40.465351 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:40.466669 (+ 1318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:40.466670 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:40.468026 (+ 1356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:40.468028 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:40.468932 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:40.468933 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:40.468991 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:40.468994 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 03:17:40.469088 (+ 94us) client_negotiation.cc:240] Negotiation successful
0929 03:17:40.469130 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":386,"mutex_wait_us":9,"thread_start_us":87,"threads_started":1}
I20260929 03:17:40.469888 23203 tablet_service.cc:1467] Processing CreateTablet for tablet 16dcef6c559c40eaab699b371c1ae99c (DEFAULT_TABLE table=test-table [id=148299c5a13c4a75af4c315c0ee3916b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:17:40.470105 23063 tablet_service.cc:1467] Processing CreateTablet for tablet 16dcef6c559c40eaab699b371c1ae99c (DEFAULT_TABLE table=test-table [id=148299c5a13c4a75af4c315c0ee3916b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:17:40.470260 23430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:40.464874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49072 (local address 127.20.178.67:35231)
0929 03:17:40.465061 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:40.465067 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:40.465079 (+ 12us) server_negotiation.cc:407] Connection header received
0929 03:17:40.465118 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:40.465120 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:40.465145 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:40.465237 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:17:40.465462 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:40.466274 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:40.467847 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:40.468612 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:40.468721 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:40.468789 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0929 03:17:40.470085 (+ 1296us) server_negotiation.cc:299] Negotiation successful
0929 03:17:40.470130 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":88,"threads_started":1}
I20260929 03:17:40.470271 23203 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 16dcef6c559c40eaab699b371c1ae99c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:17:40.470458 23063 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 16dcef6c559c40eaab699b371c1ae99c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:17:40.470832 23341 tablet_service.cc:1467] Processing CreateTablet for tablet 16dcef6c559c40eaab699b371c1ae99c (DEFAULT_TABLE table=test-table [id=148299c5a13c4a75af4c315c0ee3916b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:17:40.471127 23341 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 16dcef6c559c40eaab699b371c1ae99c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:17:40.473066 23432 tablet_bootstrap.cc:492] T 16dcef6c559c40eaab699b371c1ae99c P d68a40cd55a24edbb306ac354251f160: Bootstrap starting.
I20260929 03:17:40.473616 23432 tablet_bootstrap.cc:654] T 16dcef6c559c40eaab699b371c1ae99c P d68a40cd55a24edbb306ac354251f160: Neither blocks nor log segments found. Creating new log.
I20260929 03:17:40.473843 23432 log.cc:824] T 16dcef6c559c40eaab699b371c1ae99c P d68a40cd55a24edbb306ac354251f160: Log is configured to *not* fsync() on all Append() calls
I20260929 03:17:40.474061 23433 tablet_bootstrap.cc:492] T 16dcef6c559c40eaab699b371c1ae99c P 7502a5d6be1646cba535265e54b2f07e: Bootstrap starting.
I20260929 03:17:40.474506 23435 tablet_bootstrap.cc:492] T 16dcef6c559c40eaab699b371c1ae99c P f87970e42bbb4b10884b1606288565ca: Bootstrap starting.
I20260929 03:17:40.474655 23433 tablet_bootstrap.cc:654] T 16dcef6c559c40eaab699b371c1ae99c P 7502a5d6be1646cba535265e54b2f07e: Neither blocks nor log segments found. Creating new log.
I20260929 03:17:40.474747 23432 tablet_bootstrap.cc:492] T 16dcef6c559c40eaab699b371c1ae99c P d68a40cd55a24edbb306ac354251f160: No bootstrap required, opened a new log
I20260929 03:17:40.474822 23432 ts_tablet_manager.cc:1397] T 16dcef6c559c40eaab699b371c1ae99c P d68a40cd55a24edbb306ac354251f160: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 03:17:40.474987 23433 log.cc:824] T 16dcef6c559c40eaab699b371c1ae99c P 7502a5d6be1646cba535265e54b2f07e: Log is configured to *not* fsync() on all Append() calls
I20260929 03:17:40.475414 23435 tablet_bootstrap.cc:654] T 16dcef6c559c40eaab699b371c1ae99c P f87970e42bbb4b10884b1606288565ca: Neither blocks nor log segments found. Creating new log.
I20260929 03:17:40.475699 23435 log.cc:824] T 16dcef6c559c40eaab699b371c1ae99c P f87970e42bbb4b10884b1606288565ca: Log is configured to *not* fsync() on all Append() calls
I20260929 03:17:40.475747 23433 tablet_bootstrap.cc:492] T 16dcef6c559c40eaab699b371c1ae99c P 7502a5d6be1646cba535265e54b2f07e: No bootstrap required, opened a new log
I20260929 03:17:40.475814 23433 ts_tablet_manager.cc:1397] T 16dcef6c559c40eaab699b371c1ae99c P 7502a5d6be1646cba535265e54b2f07e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 03:17:40.476351 23435 tablet_bootstrap.cc:492] T 16dcef6c559c40eaab699b371c1ae99c P f87970e42bbb4b10884b1606288565ca: No bootstrap required, opened a new log
I20260929 03:17:40.476414 23435 ts_tablet_manager.cc:1397] T 16dcef6c559c40eaab699b371c1ae99c P f87970e42bbb4b10884b1606288565ca: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 03:17:40.476943 23432 raft_consensus.cc:348] T 16dcef6c559c40eaab699b371c1ae99c P d68a40cd55a24edbb306ac354251f160 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f87970e42bbb4b10884b1606288565ca" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 35231 } } peers { permanent_uuid: "7502a5d6be1646cba535265e54b2f07e" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 43333 } } peers { permanent_uuid: "d68a40cd55a24edbb306ac354251f160" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 34631 } }
I20260929 03:17:40.477115 23432 raft_consensus.cc:374] T 16dcef6c559c40eaab699b371c1ae99c P d68a40cd55a24edbb306ac354251f160 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:17:40.477164 23432 raft_consensus.cc:729] T 16dcef6c559c40eaab699b371c1ae99c P d68a40cd55a24edbb306ac354251f160 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d68a40cd55a24edbb306ac354251f160, State: Initialized, Role: FOLLOWER
I20260929 03:17:40.477286 23432 consensus_queue.cc:260] T 16dcef6c559c40eaab699b371c1ae99c P d68a40cd55a24edbb306ac354251f160 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f87970e42bbb4b10884b1606288565ca" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 35231 } } peers { permanent_uuid: "7502a5d6be1646cba535265e54b2f07e" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 43333 } } peers { permanent_uuid: "d68a40cd55a24edbb306ac354251f160" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 34631 } }
I20260929 03:17:40.477540 23432 ts_tablet_manager.cc:1428] T 16dcef6c559c40eaab699b371c1ae99c P d68a40cd55a24edbb306ac354251f160: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260929 03:17:40.477680 23269 heartbeater.cc:503] Master 127.20.178.126:40307 was elected leader, sending a full tablet report...
I20260929 03:17:40.477926 23433 raft_consensus.cc:348] T 16dcef6c559c40eaab699b371c1ae99c P 7502a5d6be1646cba535265e54b2f07e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f87970e42bbb4b10884b1606288565ca" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 35231 } } peers { permanent_uuid: "7502a5d6be1646cba535265e54b2f07e" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 43333 } } peers { permanent_uuid: "d68a40cd55a24edbb306ac354251f160" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 34631 } }
I20260929 03:17:40.478087 23433 raft_consensus.cc:374] T 16dcef6c559c40eaab699b371c1ae99c P 7502a5d6be1646cba535265e54b2f07e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:17:40.478124 23433 raft_consensus.cc:729] T 16dcef6c559c40eaab699b371c1ae99c P 7502a5d6be1646cba535265e54b2f07e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7502a5d6be1646cba535265e54b2f07e, State: Initialized, Role: FOLLOWER
I20260929 03:17:40.478227 23433 consensus_queue.cc:260] T 16dcef6c559c40eaab699b371c1ae99c P 7502a5d6be1646cba535265e54b2f07e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f87970e42bbb4b10884b1606288565ca" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 35231 } } peers { permanent_uuid: "7502a5d6be1646cba535265e54b2f07e" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 43333 } } peers { permanent_uuid: "d68a40cd55a24edbb306ac354251f160" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 34631 } }
I20260929 03:17:40.478289 23435 raft_consensus.cc:348] T 16dcef6c559c40eaab699b371c1ae99c P f87970e42bbb4b10884b1606288565ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f87970e42bbb4b10884b1606288565ca" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 35231 } } peers { permanent_uuid: "7502a5d6be1646cba535265e54b2f07e" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 43333 } } peers { permanent_uuid: "d68a40cd55a24edbb306ac354251f160" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 34631 } }
I20260929 03:17:40.478456 23435 raft_consensus.cc:374] T 16dcef6c559c40eaab699b371c1ae99c P f87970e42bbb4b10884b1606288565ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:17:40.478479 23433 ts_tablet_manager.cc:1428] T 16dcef6c559c40eaab699b371c1ae99c P 7502a5d6be1646cba535265e54b2f07e: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260929 03:17:40.478497 23435 raft_consensus.cc:729] T 16dcef6c559c40eaab699b371c1ae99c P f87970e42bbb4b10884b1606288565ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f87970e42bbb4b10884b1606288565ca, State: Initialized, Role: FOLLOWER
I20260929 03:17:40.478598 23129 heartbeater.cc:503] Master 127.20.178.126:40307 was elected leader, sending a full tablet report...
I20260929 03:17:40.478598 23435 consensus_queue.cc:260] T 16dcef6c559c40eaab699b371c1ae99c P f87970e42bbb4b10884b1606288565ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f87970e42bbb4b10884b1606288565ca" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 35231 } } peers { permanent_uuid: "7502a5d6be1646cba535265e54b2f07e" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 43333 } } peers { permanent_uuid: "d68a40cd55a24edbb306ac354251f160" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 34631 } }
I20260929 03:17:40.478879 23435 ts_tablet_manager.cc:1428] T 16dcef6c559c40eaab699b371c1ae99c P f87970e42bbb4b10884b1606288565ca: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260929 03:17:40.478964 23409 heartbeater.cc:503] Master 127.20.178.126:40307 was elected leader, sending a full tablet report...
W20260929 03:17:40.517117 23270 tablet.cc:2369] T 16dcef6c559c40eaab699b371c1ae99c P d68a40cd55a24edbb306ac354251f160: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 03:17:40.597925 23130 tablet.cc:2369] T 16dcef6c559c40eaab699b371c1ae99c P 7502a5d6be1646cba535265e54b2f07e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 03:17:40.672816 23410 tablet.cc:2369] T 16dcef6c559c40eaab699b371c1ae99c P f87970e42bbb4b10884b1606288565ca: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 03:17:40.703114 23439 raft_consensus.cc:482] T 16dcef6c559c40eaab699b371c1ae99c P 7502a5d6be1646cba535265e54b2f07e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:17:40.703276 23439 raft_consensus.cc:504] T 16dcef6c559c40eaab699b371c1ae99c P 7502a5d6be1646cba535265e54b2f07e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f87970e42bbb4b10884b1606288565ca" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 35231 } } peers { permanent_uuid: "7502a5d6be1646cba535265e54b2f07e" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 43333 } } peers { permanent_uuid: "d68a40cd55a24edbb306ac354251f160" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 34631 } }
I20260929 03:17:40.703684 23439 leader_election.cc:288] T 16dcef6c559c40eaab699b371c1ae99c P 7502a5d6be1646cba535265e54b2f07e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f87970e42bbb4b10884b1606288565ca (127.20.178.67:35231), d68a40cd55a24edbb306ac354251f160 (127.20.178.66:34631)
I20260929 03:17:40.707664 23431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:40.703984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.65:53255 (local address 127.20.178.66:34631)
0929 03:17:40.704031 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:40.704033 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:40.704157 (+ 124us) server_negotiation.cc:407] Connection header received
0929 03:17:40.704182 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:40.704183 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:40.704192 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:40.704241 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:17:40.704423 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:40.705204 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:40.706730 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:40.707285 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:40.707374 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:40.707382 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0929 03:17:40.707556 (+ 174us) server_negotiation.cc:299] Negotiation successful
0929 03:17:40.707590 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260929 03:17:40.707664 23430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:40.703959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.65:60619 (local address 127.20.178.67:35231)
0929 03:17:40.704002 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:40.704005 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:40.704082 (+ 77us) server_negotiation.cc:407] Connection header received
0929 03:17:40.704160 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:40.704161 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:40.704173 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:40.704232 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:17:40.704422 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:40.705197 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:40.706729 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:40.707296 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:40.707374 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:40.707382 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0929 03:17:40.707550 (+ 168us) server_negotiation.cc:299] Negotiation successful
0929 03:17:40.707590 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260929 03:17:40.707747 23441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:40.703872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.67:35231 (local address 127.20.178.65:60619)
0929 03:17:40.703998 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0929 03:17:40.704011 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 03:17:40.704075 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:17:40.704277 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:17:40.704277 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:17:40.704279 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:17:40.704346 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:40.704347 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:40.705292 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:40.705293 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:40.706624 (+ 1331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:40.706627 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:40.707388 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:40.707389 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:40.707466 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:40.707471 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 03:17:40.707531 (+ 60us) client_negotiation.cc:240] Negotiation successful
0929 03:17:40.707586 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"mutex_wait_us":56,"thread_start_us":60,"threads_started":1}
I20260929 03:17:40.707755 23442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:40.703920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.66:34631 (local address 127.20.178.65:53255)
0929 03:17:40.704054 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0929 03:17:40.704063 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 03:17:40.704120 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:17:40.704261 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:17:40.704262 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:17:40.704265 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:17:40.704330 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:40.704332 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:40.705292 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:40.705293 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:40.706625 (+ 1332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:40.706627 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:40.707387 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:40.707389 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:40.707466 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:40.707471 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 03:17:40.707531 (+ 60us) client_negotiation.cc:240] Negotiation successful
0929 03:17:40.707586 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"mutex_wait_us":87,"thread_start_us":74,"threads_started":1}
I20260929 03:17:40.708118 23363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16dcef6c559c40eaab699b371c1ae99c" candidate_uuid: "7502a5d6be1646cba535265e54b2f07e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f87970e42bbb4b10884b1606288565ca" is_pre_election: true
I20260929 03:17:40.708302 23363 raft_consensus.cc:2454] T 16dcef6c559c40eaab699b371c1ae99c P f87970e42bbb4b10884b1606288565ca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7502a5d6be1646cba535265e54b2f07e in term 0.
I20260929 03:17:40.708313 23223 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16dcef6c559c40eaab699b371c1ae99c" candidate_uuid: "7502a5d6be1646cba535265e54b2f07e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d68a40cd55a24edbb306ac354251f160" is_pre_election: true
I20260929 03:17:40.708478 23223 raft_consensus.cc:2454] T 16dcef6c559c40eaab699b371c1ae99c P d68a40cd55a24edbb306ac354251f160 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7502a5d6be1646cba535265e54b2f07e in term 0.
I20260929 03:17:40.708539 23016 leader_election.cc:302] T 16dcef6c559c40eaab699b371c1ae99c P 7502a5d6be1646cba535265e54b2f07e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7502a5d6be1646cba535265e54b2f07e, f87970e42bbb4b10884b1606288565ca; no voters:
I20260929 03:17:40.708709 23439 raft_consensus.cc:2781] T 16dcef6c559c40eaab699b371c1ae99c P 7502a5d6be1646cba535265e54b2f07e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 03:17:40.708792 23439 raft_consensus.cc:482] T 16dcef6c559c40eaab699b371c1ae99c P 7502a5d6be1646cba535265e54b2f07e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 03:17:40.708829 23439 raft_consensus.cc:3037] T 16dcef6c559c40eaab699b371c1ae99c P 7502a5d6be1646cba535265e54b2f07e [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:17:40.709508 23439 raft_consensus.cc:504] T 16dcef6c559c40eaab699b371c1ae99c P 7502a5d6be1646cba535265e54b2f07e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f87970e42bbb4b10884b1606288565ca" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 35231 } } peers { permanent_uuid: "7502a5d6be1646cba535265e54b2f07e" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 43333 } } peers { permanent_uuid: "d68a40cd55a24edbb306ac354251f160" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 34631 } }
I20260929 03:17:40.709666 23439 leader_election.cc:288] T 16dcef6c559c40eaab699b371c1ae99c P 7502a5d6be1646cba535265e54b2f07e [CANDIDATE]: Term 1 election: Requested vote from peers f87970e42bbb4b10884b1606288565ca (127.20.178.67:35231), d68a40cd55a24edbb306ac354251f160 (127.20.178.66:34631)
I20260929 03:17:40.709817 23223 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16dcef6c559c40eaab699b371c1ae99c" candidate_uuid: "7502a5d6be1646cba535265e54b2f07e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d68a40cd55a24edbb306ac354251f160"
I20260929 03:17:40.709846 23363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16dcef6c559c40eaab699b371c1ae99c" candidate_uuid: "7502a5d6be1646cba535265e54b2f07e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f87970e42bbb4b10884b1606288565ca"
I20260929 03:17:40.709892 23223 raft_consensus.cc:3037] T 16dcef6c559c40eaab699b371c1ae99c P d68a40cd55a24edbb306ac354251f160 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:17:40.709935 23363 raft_consensus.cc:3037] T 16dcef6c559c40eaab699b371c1ae99c P f87970e42bbb4b10884b1606288565ca [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:17:40.710561 23223 raft_consensus.cc:2454] T 16dcef6c559c40eaab699b371c1ae99c P d68a40cd55a24edbb306ac354251f160 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7502a5d6be1646cba535265e54b2f07e in term 1.
I20260929 03:17:40.710723 23014 leader_election.cc:302] T 16dcef6c559c40eaab699b371c1ae99c P 7502a5d6be1646cba535265e54b2f07e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7502a5d6be1646cba535265e54b2f07e, d68a40cd55a24edbb306ac354251f160; no voters:
I20260929 03:17:40.710738 23363 raft_consensus.cc:2454] T 16dcef6c559c40eaab699b371c1ae99c P f87970e42bbb4b10884b1606288565ca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7502a5d6be1646cba535265e54b2f07e in term 1.
I20260929 03:17:40.710804 23439 raft_consensus.cc:2781] T 16dcef6c559c40eaab699b371c1ae99c P 7502a5d6be1646cba535265e54b2f07e [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:17:40.710932 23439 raft_consensus.cc:686] T 16dcef6c559c40eaab699b371c1ae99c P 7502a5d6be1646cba535265e54b2f07e [term 1 LEADER]: Becoming Leader. State: Replica: 7502a5d6be1646cba535265e54b2f07e, State: Running, Role: LEADER
I20260929 03:17:40.711053 23439 consensus_queue.cc:237] T 16dcef6c559c40eaab699b371c1ae99c P 7502a5d6be1646cba535265e54b2f07e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f87970e42bbb4b10884b1606288565ca" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 35231 } } peers { permanent_uuid: "7502a5d6be1646cba535265e54b2f07e" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 43333 } } peers { permanent_uuid: "d68a40cd55a24edbb306ac354251f160" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 34631 } }
I20260929 03:17:40.711879 22926 catalog_manager.cc:5495] T 16dcef6c559c40eaab699b371c1ae99c P 7502a5d6be1646cba535265e54b2f07e reported cstate change: term changed from 0 to 1, leader changed from <none> to 7502a5d6be1646cba535265e54b2f07e (127.20.178.65). New cstate: current_term: 1 leader_uuid: "7502a5d6be1646cba535265e54b2f07e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f87970e42bbb4b10884b1606288565ca" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 35231 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7502a5d6be1646cba535265e54b2f07e" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 43333 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d68a40cd55a24edbb306ac354251f160" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 34631 } health_report { overall_health: UNKNOWN } } }
Sep 29 03:17:40 dist-test-slave-c9qh krb5kdc[22881](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651859, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.126@KRBTEST.COM for kudu/127.20.178.126@KRBTEST.COM
I20260929 03:17:40.767926 22975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:40.761419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37162 (local address 127.20.178.126:40307)
0929 03:17:40.761510 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:40.761512 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:40.761617 (+ 105us) server_negotiation.cc:407] Connection header received
0929 03:17:40.761666 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:40.761667 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:40.761676 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:40.761716 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:17:40.762147 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:40.762615 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:40.763293 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:40.763468 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:40.763516 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:40.765546 (+ 2030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:17:40.765547 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:17:40.765548 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:17:40.765554 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:17:40.766750 (+ 1196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:40.767126 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:40.767127 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:40.767127 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:40.767179 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:40.767366 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:40.767367 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:40.767368 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:40.767471 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:17:40.767546 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0929 03:17:40.767694 (+ 148us) server_negotiation.cc:299] Negotiation successful
0929 03:17:40.767820 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":6,"server-negotiator.queue_time_us":62}
I20260929 03:17:40.768021 23450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:40.761326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:40307 (local address 127.0.0.1:37162)
0929 03:17:40.761488 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0929 03:17:40.761499 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 03:17:40.761613 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:17:40.761735 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:17:40.761735 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:17:40.762037 (+ 302us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:17:40.762083 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:40.762085 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:40.762699 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:40.762700 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:40.763219 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:40.763221 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:40.763535 (+ 314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:40.763536 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:40.763578 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:40.764068 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:17:40.764072 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:17:40.765454 (+ 1382us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:17:40.766847 (+ 1393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:17:40.766854 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:17:40.766856 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:17:40.767044 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:17:40.767249 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:17:40.767250 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:17:40.767250 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:17:40.767305 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:17:40.767537 (+ 232us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:17:40.767544 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:17:40.767588 (+ 44us) client_negotiation.cc:769] Sending connection context
0929 03:17:40.767671 (+ 83us) client_negotiation.cc:240] Negotiation successful
0929 03:17:40.767878 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"mutex_wait_us":102,"thread_start_us":71,"threads_started":1}
I20260929 03:17:40.770094 22926 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.20.178.126@KRBTEST.COM'} at 127.0.0.1:37162:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260929 03:17:40.770323 22926 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 03:17:40.771880 23341 tablet_service.cc:1467] Processing CreateTablet for tablet 9d9faa428bf449a8bc4c1594978ded91 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=cb75ef9694474964afdf022c9db3eee3]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 03:17:40.771871 23063 tablet_service.cc:1467] Processing CreateTablet for tablet 9d9faa428bf449a8bc4c1594978ded91 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=cb75ef9694474964afdf022c9db3eee3]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 03:17:40.771883 23203 tablet_service.cc:1467] Processing CreateTablet for tablet 9d9faa428bf449a8bc4c1594978ded91 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=cb75ef9694474964afdf022c9db3eee3]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 03:17:40.772037 23063 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d9faa428bf449a8bc4c1594978ded91. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:17:40.772037 23341 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d9faa428bf449a8bc4c1594978ded91. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:17:40.772055 23203 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d9faa428bf449a8bc4c1594978ded91. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:17:40.773298 23432 tablet_bootstrap.cc:492] T 9d9faa428bf449a8bc4c1594978ded91 P d68a40cd55a24edbb306ac354251f160: Bootstrap starting.
I20260929 03:17:40.773599 23435 tablet_bootstrap.cc:492] T 9d9faa428bf449a8bc4c1594978ded91 P f87970e42bbb4b10884b1606288565ca: Bootstrap starting.
I20260929 03:17:40.773748 23432 tablet_bootstrap.cc:654] T 9d9faa428bf449a8bc4c1594978ded91 P d68a40cd55a24edbb306ac354251f160: Neither blocks nor log segments found. Creating new log.
I20260929 03:17:40.774176 23435 tablet_bootstrap.cc:654] T 9d9faa428bf449a8bc4c1594978ded91 P f87970e42bbb4b10884b1606288565ca: Neither blocks nor log segments found. Creating new log.
I20260929 03:17:40.774456 23432 tablet_bootstrap.cc:492] T 9d9faa428bf449a8bc4c1594978ded91 P d68a40cd55a24edbb306ac354251f160: No bootstrap required, opened a new log
I20260929 03:17:40.774514 23432 ts_tablet_manager.cc:1397] T 9d9faa428bf449a8bc4c1594978ded91 P d68a40cd55a24edbb306ac354251f160: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260929 03:17:40.774540 23433 tablet_bootstrap.cc:492] T 9d9faa428bf449a8bc4c1594978ded91 P 7502a5d6be1646cba535265e54b2f07e: Bootstrap starting.
I20260929 03:17:40.774650 23432 raft_consensus.cc:348] T 9d9faa428bf449a8bc4c1594978ded91 P d68a40cd55a24edbb306ac354251f160 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7502a5d6be1646cba535265e54b2f07e" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 43333 } } peers { permanent_uuid: "d68a40cd55a24edbb306ac354251f160" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 34631 } } peers { permanent_uuid: "f87970e42bbb4b10884b1606288565ca" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 35231 } }
I20260929 03:17:40.774740 23432 raft_consensus.cc:374] T 9d9faa428bf449a8bc4c1594978ded91 P d68a40cd55a24edbb306ac354251f160 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:17:40.774770 23432 raft_consensus.cc:729] T 9d9faa428bf449a8bc4c1594978ded91 P d68a40cd55a24edbb306ac354251f160 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d68a40cd55a24edbb306ac354251f160, State: Initialized, Role: FOLLOWER
I20260929 03:17:40.774816 23432 consensus_queue.cc:260] T 9d9faa428bf449a8bc4c1594978ded91 P d68a40cd55a24edbb306ac354251f160 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7502a5d6be1646cba535265e54b2f07e" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 43333 } } peers { permanent_uuid: "d68a40cd55a24edbb306ac354251f160" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 34631 } } peers { permanent_uuid: "f87970e42bbb4b10884b1606288565ca" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 35231 } }
I20260929 03:17:40.774852 23435 tablet_bootstrap.cc:492] T 9d9faa428bf449a8bc4c1594978ded91 P f87970e42bbb4b10884b1606288565ca: No bootstrap required, opened a new log
I20260929 03:17:40.774899 23435 ts_tablet_manager.cc:1397] T 9d9faa428bf449a8bc4c1594978ded91 P f87970e42bbb4b10884b1606288565ca: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260929 03:17:40.774912 23432 ts_tablet_manager.cc:1428] T 9d9faa428bf449a8bc4c1594978ded91 P d68a40cd55a24edbb306ac354251f160: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260929 03:17:40.774966 23433 tablet_bootstrap.cc:654] T 9d9faa428bf449a8bc4c1594978ded91 P 7502a5d6be1646cba535265e54b2f07e: Neither blocks nor log segments found. Creating new log.
I20260929 03:17:40.775069 23438 tablet_replica.cc:439] T 9d9faa428bf449a8bc4c1594978ded91 P d68a40cd55a24edbb306ac354251f160: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7502a5d6be1646cba535265e54b2f07e" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 43333 } } peers { permanent_uuid: "d68a40cd55a24edbb306ac354251f160" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 34631 } } peers { permanent_uuid: "f87970e42bbb4b10884b1606288565ca" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 35231 } } }
I20260929 03:17:40.775090 23435 raft_consensus.cc:348] T 9d9faa428bf449a8bc4c1594978ded91 P f87970e42bbb4b10884b1606288565ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7502a5d6be1646cba535265e54b2f07e" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 43333 } } peers { permanent_uuid: "d68a40cd55a24edbb306ac354251f160" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 34631 } } peers { permanent_uuid: "f87970e42bbb4b10884b1606288565ca" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 35231 } }
I20260929 03:17:40.775169 23435 raft_consensus.cc:374] T 9d9faa428bf449a8bc4c1594978ded91 P f87970e42bbb4b10884b1606288565ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:17:40.775166 23438 tablet_replica.cc:442] T 9d9faa428bf449a8bc4c1594978ded91 P d68a40cd55a24edbb306ac354251f160: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 03:17:40.775195 23435 raft_consensus.cc:729] T 9d9faa428bf449a8bc4c1594978ded91 P f87970e42bbb4b10884b1606288565ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f87970e42bbb4b10884b1606288565ca, State: Initialized, Role: FOLLOWER
I20260929 03:17:40.775239 23435 consensus_queue.cc:260] T 9d9faa428bf449a8bc4c1594978ded91 P f87970e42bbb4b10884b1606288565ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7502a5d6be1646cba535265e54b2f07e" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 43333 } } peers { permanent_uuid: "d68a40cd55a24edbb306ac354251f160" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 34631 } } peers { permanent_uuid: "f87970e42bbb4b10884b1606288565ca" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 35231 } }
I20260929 03:17:40.775326 23435 ts_tablet_manager.cc:1428] T 9d9faa428bf449a8bc4c1594978ded91 P f87970e42bbb4b10884b1606288565ca: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260929 03:17:40.775460 23440 tablet_replica.cc:439] T 9d9faa428bf449a8bc4c1594978ded91 P f87970e42bbb4b10884b1606288565ca: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7502a5d6be1646cba535265e54b2f07e" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 43333 } } peers { permanent_uuid: "d68a40cd55a24edbb306ac354251f160" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 34631 } } peers { permanent_uuid: "f87970e42bbb4b10884b1606288565ca" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 35231 } } }
I20260929 03:17:40.775542 23440 tablet_replica.cc:442] T 9d9faa428bf449a8bc4c1594978ded91 P f87970e42bbb4b10884b1606288565ca: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 03:17:40.775877 23433 tablet_bootstrap.cc:492] T 9d9faa428bf449a8bc4c1594978ded91 P 7502a5d6be1646cba535265e54b2f07e: No bootstrap required, opened a new log
I20260929 03:17:40.775928 23433 ts_tablet_manager.cc:1397] T 9d9faa428bf449a8bc4c1594978ded91 P 7502a5d6be1646cba535265e54b2f07e: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260929 03:17:40.776057 23433 raft_consensus.cc:348] T 9d9faa428bf449a8bc4c1594978ded91 P 7502a5d6be1646cba535265e54b2f07e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7502a5d6be1646cba535265e54b2f07e" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 43333 } } peers { permanent_uuid: "d68a40cd55a24edbb306ac354251f160" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 34631 } } peers { permanent_uuid: "f87970e42bbb4b10884b1606288565ca" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 35231 } }
I20260929 03:17:40.776122 23433 raft_consensus.cc:374] T 9d9faa428bf449a8bc4c1594978ded91 P 7502a5d6be1646cba535265e54b2f07e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:17:40.776145 23433 raft_consensus.cc:729] T 9d9faa428bf449a8bc4c1594978ded91 P 7502a5d6be1646cba535265e54b2f07e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7502a5d6be1646cba535265e54b2f07e, State: Initialized, Role: FOLLOWER
I20260929 03:17:40.776197 23433 consensus_queue.cc:260] T 9d9faa428bf449a8bc4c1594978ded91 P 7502a5d6be1646cba535265e54b2f07e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7502a5d6be1646cba535265e54b2f07e" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 43333 } } peers { permanent_uuid: "d68a40cd55a24edbb306ac354251f160" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 34631 } } peers { permanent_uuid: "f87970e42bbb4b10884b1606288565ca" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 35231 } }
I20260929 03:17:40.776295 23433 ts_tablet_manager.cc:1428] T 9d9faa428bf449a8bc4c1594978ded91 P 7502a5d6be1646cba535265e54b2f07e: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260929 03:17:40.776333 23439 tablet_replica.cc:439] T 9d9faa428bf449a8bc4c1594978ded91 P 7502a5d6be1646cba535265e54b2f07e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7502a5d6be1646cba535265e54b2f07e" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 43333 } } peers { permanent_uuid: "d68a40cd55a24edbb306ac354251f160" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 34631 } } peers { permanent_uuid: "f87970e42bbb4b10884b1606288565ca" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 35231 } } }
I20260929 03:17:40.776423 23439 tablet_replica.cc:442] T 9d9faa428bf449a8bc4c1594978ded91 P 7502a5d6be1646cba535265e54b2f07e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 03:17:40.969828 23438 raft_consensus.cc:482] T 9d9faa428bf449a8bc4c1594978ded91 P d68a40cd55a24edbb306ac354251f160 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:17:40.969944 23438 raft_consensus.cc:504] T 9d9faa428bf449a8bc4c1594978ded91 P d68a40cd55a24edbb306ac354251f160 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7502a5d6be1646cba535265e54b2f07e" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 43333 } } peers { permanent_uuid: "d68a40cd55a24edbb306ac354251f160" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 34631 } } peers { permanent_uuid: "f87970e42bbb4b10884b1606288565ca" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 35231 } }
I20260929 03:17:40.970366 23438 leader_election.cc:288] T 9d9faa428bf449a8bc4c1594978ded91 P d68a40cd55a24edbb306ac354251f160 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7502a5d6be1646cba535265e54b2f07e (127.20.178.65:43333), f87970e42bbb4b10884b1606288565ca (127.20.178.67:35231)
I20260929 03:17:40.973342 23430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:40.970566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.66:42909 (local address 127.20.178.67:35231)
0929 03:17:40.970613 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:40.970615 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:40.970712 (+ 97us) server_negotiation.cc:407] Connection header received
0929 03:17:40.970749 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:40.970750 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:40.970760 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:40.970848 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:17:40.971002 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:40.971479 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:40.972547 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:40.972975 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:40.973078 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:40.973088 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0929 03:17:40.973238 (+ 150us) server_negotiation.cc:299] Negotiation successful
0929 03:17:40.973279 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260929 03:17:40.973356 23455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:40.970515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.67:35231 (local address 127.20.178.66:42909)
0929 03:17:40.970631 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0929 03:17:40.970641 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 03:17:40.970691 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:17:40.970853 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:17:40.970854 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:17:40.970856 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:17:40.970926 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:40.970928 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:40.971564 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:40.971565 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:40.972344 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:40.972346 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:40.973079 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:40.973080 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:40.973149 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:40.973153 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 03:17:40.973228 (+ 75us) client_negotiation.cc:240] Negotiation successful
0929 03:17:40.973269 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":54,"threads_started":1}
I20260929 03:17:40.973469 23454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:40.970513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.65:43333 (local address 127.20.178.66:56641)
0929 03:17:40.970608 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0929 03:17:40.970621 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 03:17:40.970684 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:17:40.970959 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:17:40.970960 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:17:40.970962 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:17:40.971023 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:40.971025 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:40.971841 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:40.971842 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:40.972567 (+ 725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:40.972569 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:40.973186 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:40.973187 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:40.973247 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:40.973250 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 03:17:40.973311 (+ 61us) client_negotiation.cc:240] Negotiation successful
0929 03:17:40.973355 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":77,"mutex_wait_us":1,"thread_start_us":56,"threads_started":1}
I20260929 03:17:40.973546 23456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:40.970666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.66:56641 (local address 127.20.178.65:43333)
0929 03:17:40.970834 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:40.970837 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:40.970845 (+ 8us) server_negotiation.cc:407] Connection header received
0929 03:17:40.970871 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:40.970872 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:40.970881 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:40.970957 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:17:40.971104 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:40.971589 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:40.972696 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:40.973072 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:40.973158 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:40.973166 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0929 03:17:40.973330 (+ 164us) server_negotiation.cc:299] Negotiation successful
0929 03:17:40.973369 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":61,"threads_started":1}
I20260929 03:17:40.973816 23363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d9faa428bf449a8bc4c1594978ded91" candidate_uuid: "d68a40cd55a24edbb306ac354251f160" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f87970e42bbb4b10884b1606288565ca" is_pre_election: true
I20260929 03:17:40.973933 23363 raft_consensus.cc:2454] T 9d9faa428bf449a8bc4c1594978ded91 P f87970e42bbb4b10884b1606288565ca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d68a40cd55a24edbb306ac354251f160 in term 0.
I20260929 03:17:40.974068 23083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d9faa428bf449a8bc4c1594978ded91" candidate_uuid: "d68a40cd55a24edbb306ac354251f160" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7502a5d6be1646cba535265e54b2f07e" is_pre_election: true
I20260929 03:17:40.974098 23156 leader_election.cc:302] T 9d9faa428bf449a8bc4c1594978ded91 P d68a40cd55a24edbb306ac354251f160 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d68a40cd55a24edbb306ac354251f160, f87970e42bbb4b10884b1606288565ca; no voters:
I20260929 03:17:40.974188 23083 raft_consensus.cc:2454] T 9d9faa428bf449a8bc4c1594978ded91 P 7502a5d6be1646cba535265e54b2f07e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d68a40cd55a24edbb306ac354251f160 in term 0.
I20260929 03:17:40.974255 23438 raft_consensus.cc:2781] T 9d9faa428bf449a8bc4c1594978ded91 P d68a40cd55a24edbb306ac354251f160 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 03:17:40.974299 23438 raft_consensus.cc:482] T 9d9faa428bf449a8bc4c1594978ded91 P d68a40cd55a24edbb306ac354251f160 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 03:17:40.974324 23438 raft_consensus.cc:3037] T 9d9faa428bf449a8bc4c1594978ded91 P d68a40cd55a24edbb306ac354251f160 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:17:40.975044 23438 raft_consensus.cc:504] T 9d9faa428bf449a8bc4c1594978ded91 P d68a40cd55a24edbb306ac354251f160 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7502a5d6be1646cba535265e54b2f07e" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 43333 } } peers { permanent_uuid: "d68a40cd55a24edbb306ac354251f160" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 34631 } } peers { permanent_uuid: "f87970e42bbb4b10884b1606288565ca" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 35231 } }
I20260929 03:17:40.975189 23438 leader_election.cc:288] T 9d9faa428bf449a8bc4c1594978ded91 P d68a40cd55a24edbb306ac354251f160 [CANDIDATE]: Term 1 election: Requested vote from peers 7502a5d6be1646cba535265e54b2f07e (127.20.178.65:43333), f87970e42bbb4b10884b1606288565ca (127.20.178.67:35231)
I20260929 03:17:40.975324 23083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d9faa428bf449a8bc4c1594978ded91" candidate_uuid: "d68a40cd55a24edbb306ac354251f160" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7502a5d6be1646cba535265e54b2f07e"
I20260929 03:17:40.975401 23083 raft_consensus.cc:3037] T 9d9faa428bf449a8bc4c1594978ded91 P 7502a5d6be1646cba535265e54b2f07e [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:17:40.975360 23363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d9faa428bf449a8bc4c1594978ded91" candidate_uuid: "d68a40cd55a24edbb306ac354251f160" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f87970e42bbb4b10884b1606288565ca"
I20260929 03:17:40.975449 23363 raft_consensus.cc:3037] T 9d9faa428bf449a8bc4c1594978ded91 P f87970e42bbb4b10884b1606288565ca [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:17:40.976040 23363 raft_consensus.cc:2454] T 9d9faa428bf449a8bc4c1594978ded91 P f87970e42bbb4b10884b1606288565ca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d68a40cd55a24edbb306ac354251f160 in term 1.
I20260929 03:17:40.976106 23083 raft_consensus.cc:2454] T 9d9faa428bf449a8bc4c1594978ded91 P 7502a5d6be1646cba535265e54b2f07e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d68a40cd55a24edbb306ac354251f160 in term 1.
I20260929 03:17:40.976243 23156 leader_election.cc:302] T 9d9faa428bf449a8bc4c1594978ded91 P d68a40cd55a24edbb306ac354251f160 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d68a40cd55a24edbb306ac354251f160, f87970e42bbb4b10884b1606288565ca; no voters:
I20260929 03:17:40.976329 23438 raft_consensus.cc:2781] T 9d9faa428bf449a8bc4c1594978ded91 P d68a40cd55a24edbb306ac354251f160 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:17:40.976426 23438 raft_consensus.cc:686] T 9d9faa428bf449a8bc4c1594978ded91 P d68a40cd55a24edbb306ac354251f160 [term 1 LEADER]: Becoming Leader. State: Replica: d68a40cd55a24edbb306ac354251f160, State: Running, Role: LEADER
I20260929 03:17:40.976541 23438 consensus_queue.cc:237] T 9d9faa428bf449a8bc4c1594978ded91 P d68a40cd55a24edbb306ac354251f160 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7502a5d6be1646cba535265e54b2f07e" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 43333 } } peers { permanent_uuid: "d68a40cd55a24edbb306ac354251f160" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 34631 } } peers { permanent_uuid: "f87970e42bbb4b10884b1606288565ca" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 35231 } }
I20260929 03:17:40.976950 23457 tablet_replica.cc:439] T 9d9faa428bf449a8bc4c1594978ded91 P d68a40cd55a24edbb306ac354251f160: TxnStatusTablet state changed. Reason: New leader d68a40cd55a24edbb306ac354251f160. Latest consensus state: current_term: 1 leader_uuid: "d68a40cd55a24edbb306ac354251f160" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7502a5d6be1646cba535265e54b2f07e" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 43333 } } peers { permanent_uuid: "d68a40cd55a24edbb306ac354251f160" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 34631 } } peers { permanent_uuid: "f87970e42bbb4b10884b1606288565ca" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 35231 } } }
I20260929 03:17:40.977025 23457 tablet_replica.cc:442] T 9d9faa428bf449a8bc4c1594978ded91 P d68a40cd55a24edbb306ac354251f160: This TxnStatusTablet replica's current role is: LEADER
I20260929 03:17:40.977188 23459 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260929 03:17:40.977336 23459 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260929 03:17:40.977358 22925 catalog_manager.cc:5495] T 9d9faa428bf449a8bc4c1594978ded91 P d68a40cd55a24edbb306ac354251f160 reported cstate change: term changed from 0 to 1, leader changed from <none> to d68a40cd55a24edbb306ac354251f160 (127.20.178.66). New cstate: current_term: 1 leader_uuid: "d68a40cd55a24edbb306ac354251f160" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7502a5d6be1646cba535265e54b2f07e" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 43333 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d68a40cd55a24edbb306ac354251f160" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 34631 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f87970e42bbb4b10884b1606288565ca" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 35231 } health_report { overall_health: UNKNOWN } } }
I20260929 03:17:41.003756 23450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:41.000999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.66:34631 (local address 127.0.0.1:38746)
0929 03:17:41.001063 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0929 03:17:41.001076 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 03:17:41.001130 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:17:41.001314 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:17:41.001315 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:17:41.001316 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 03:17:41.001377 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:41.001379 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:41.002282 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:41.002282 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:41.002836 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:41.002837 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:41.003290 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:41.003291 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:41.003336 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:41.003350 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 03:17:41.003621 (+ 271us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 03:17:41.003628 (+ 7us) client_negotiation.cc:769] Sending connection context
0929 03:17:41.003653 (+ 25us) client_negotiation.cc:240] Negotiation successful
0929 03:17:41.003690 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20}
I20260929 03:17:41.003768 23431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:41.001076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38746 (local address 127.20.178.66:34631)
0929 03:17:41.001145 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:41.001148 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:41.001160 (+ 12us) server_negotiation.cc:407] Connection header received
0929 03:17:41.001216 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:41.001217 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:41.001238 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:41.001314 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:17:41.001419 (+ 105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:41.002203 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:41.002915 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:41.003217 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:41.003286 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:41.003437 (+ 151us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:17:41.003560 (+ 123us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:17:41.003645 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0929 03:17:41.003670 (+ 25us) server_negotiation.cc:299] Negotiation successful
0929 03:17:41.003689 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260929 03:17:41.005612 23083 raft_consensus.cc:1261] T 9d9faa428bf449a8bc4c1594978ded91 P 7502a5d6be1646cba535265e54b2f07e [term 1 FOLLOWER]: Refusing update from remote peer d68a40cd55a24edbb306ac354251f160: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:17:41.005707 23363 raft_consensus.cc:1261] T 9d9faa428bf449a8bc4c1594978ded91 P f87970e42bbb4b10884b1606288565ca [term 1 FOLLOWER]: Refusing update from remote peer d68a40cd55a24edbb306ac354251f160: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:17:41.005954 23457 consensus_queue.cc:1035] T 9d9faa428bf449a8bc4c1594978ded91 P d68a40cd55a24edbb306ac354251f160 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7502a5d6be1646cba535265e54b2f07e" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 43333 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:17:41.006109 23438 consensus_queue.cc:1035] T 9d9faa428bf449a8bc4c1594978ded91 P d68a40cd55a24edbb306ac354251f160 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f87970e42bbb4b10884b1606288565ca" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 35231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:17:41.007165 23439 tablet_replica.cc:439] T 9d9faa428bf449a8bc4c1594978ded91 P 7502a5d6be1646cba535265e54b2f07e: TxnStatusTablet state changed. Reason: New leader d68a40cd55a24edbb306ac354251f160. Latest consensus state: current_term: 1 leader_uuid: "d68a40cd55a24edbb306ac354251f160" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7502a5d6be1646cba535265e54b2f07e" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 43333 } } peers { permanent_uuid: "d68a40cd55a24edbb306ac354251f160" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 34631 } } peers { permanent_uuid: "f87970e42bbb4b10884b1606288565ca" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 35231 } } }
I20260929 03:17:41.007256 23439 tablet_replica.cc:442] T 9d9faa428bf449a8bc4c1594978ded91 P 7502a5d6be1646cba535265e54b2f07e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 03:17:41.007215 23440 tablet_replica.cc:439] T 9d9faa428bf449a8bc4c1594978ded91 P f87970e42bbb4b10884b1606288565ca: TxnStatusTablet state changed. Reason: New leader d68a40cd55a24edbb306ac354251f160. Latest consensus state: current_term: 1 leader_uuid: "d68a40cd55a24edbb306ac354251f160" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7502a5d6be1646cba535265e54b2f07e" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 43333 } } peers { permanent_uuid: "d68a40cd55a24edbb306ac354251f160" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 34631 } } peers { permanent_uuid: "f87970e42bbb4b10884b1606288565ca" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 35231 } } }
I20260929 03:17:41.007320 23440 tablet_replica.cc:442] T 9d9faa428bf449a8bc4c1594978ded91 P f87970e42bbb4b10884b1606288565ca: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 03:17:41.008136 23438 tablet_replica.cc:439] T 9d9faa428bf449a8bc4c1594978ded91 P d68a40cd55a24edbb306ac354251f160: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d68a40cd55a24edbb306ac354251f160" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7502a5d6be1646cba535265e54b2f07e" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 43333 } } peers { permanent_uuid: "d68a40cd55a24edbb306ac354251f160" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 34631 } } peers { permanent_uuid: "f87970e42bbb4b10884b1606288565ca" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 35231 } } }
I20260929 03:17:41.008217 23438 tablet_replica.cc:442] T 9d9faa428bf449a8bc4c1594978ded91 P d68a40cd55a24edbb306ac354251f160: This TxnStatusTablet replica's current role is: LEADER
I20260929 03:17:41.008260 23438 tablet_replica.cc:439] T 9d9faa428bf449a8bc4c1594978ded91 P d68a40cd55a24edbb306ac354251f160: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d68a40cd55a24edbb306ac354251f160" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7502a5d6be1646cba535265e54b2f07e" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 43333 } } peers { permanent_uuid: "d68a40cd55a24edbb306ac354251f160" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 34631 } } peers { permanent_uuid: "f87970e42bbb4b10884b1606288565ca" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 35231 } } }
I20260929 03:17:41.008316 23438 tablet_replica.cc:442] T 9d9faa428bf449a8bc4c1594978ded91 P d68a40cd55a24edbb306ac354251f160: This TxnStatusTablet replica's current role is: LEADER
I20260929 03:17:41.008534 23439 tablet_replica.cc:439] T 9d9faa428bf449a8bc4c1594978ded91 P 7502a5d6be1646cba535265e54b2f07e: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d68a40cd55a24edbb306ac354251f160" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7502a5d6be1646cba535265e54b2f07e" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 43333 } } peers { permanent_uuid: "d68a40cd55a24edbb306ac354251f160" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 34631 } } peers { permanent_uuid: "f87970e42bbb4b10884b1606288565ca" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 35231 } } }
I20260929 03:17:41.008622 23439 tablet_replica.cc:442] T 9d9faa428bf449a8bc4c1594978ded91 P 7502a5d6be1646cba535265e54b2f07e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 03:17:41.009119 23462 mvcc.cc:205] Tried to move back new op lower bound from 7334510022676447232 to 7334510022560456704. Current Snapshot: MvccSnapshot[applied={T|T < 7334510022676447232}]
I20260929 03:17:41.009264 23440 tablet_replica.cc:439] T 9d9faa428bf449a8bc4c1594978ded91 P f87970e42bbb4b10884b1606288565ca: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d68a40cd55a24edbb306ac354251f160" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7502a5d6be1646cba535265e54b2f07e" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 43333 } } peers { permanent_uuid: "d68a40cd55a24edbb306ac354251f160" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 34631 } } peers { permanent_uuid: "f87970e42bbb4b10884b1606288565ca" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 35231 } } }
I20260929 03:17:41.009356 23440 tablet_replica.cc:442] T 9d9faa428bf449a8bc4c1594978ded91 P f87970e42bbb4b10884b1606288565ca: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 03:17:41.010080 23463 mvcc.cc:205] Tried to move back new op lower bound from 7334510022676447232 to 7334510022560456704. Current Snapshot: MvccSnapshot[applied={T|T < 7334510022676447232 or (T in {7334510022676447232})}]
I20260929 03:17:41.015383 23456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:41.011052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33516 (local address 127.20.178.65:43333)
0929 03:17:41.011091 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:41.011092 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:41.012813 (+ 1721us) server_negotiation.cc:407] Connection header received
0929 03:17:41.012867 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:41.012868 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:41.012889 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:41.012958 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:17:41.013071 (+ 113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:41.013808 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:41.014534 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:41.014863 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:41.014933 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:41.015048 (+ 115us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:17:41.015182 (+ 134us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:17:41.015254 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0929 03:17:41.015290 (+ 36us) server_negotiation.cc:299] Negotiation successful
0929 03:17:41.015311 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260929 03:17:41.020432 23474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:41.017745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.66:34631 (local address 127.20.178.65:46399)
0929 03:17:41.017905 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0929 03:17:41.017917 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 03:17:41.017980 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:17:41.018110 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:17:41.018110 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:17:41.018111 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 03:17:41.018160 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:41.018161 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:41.019012 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:41.019012 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:41.019578 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:41.019579 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:41.020045 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:41.020046 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:41.020093 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:41.020106 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 03:17:41.020296 (+ 190us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 03:17:41.020297 (+ 1us) client_negotiation.cc:769] Sending connection context
0929 03:17:41.020333 (+ 36us) client_negotiation.cc:240] Negotiation successful
0929 03:17:41.020363 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":72,"threads_started":1}
I20260929 03:17:41.020426 23431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:41.017803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.65:46399 (local address 127.20.178.66:34631)
0929 03:17:41.017843 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:41.017845 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:41.017984 (+ 139us) server_negotiation.cc:407] Connection header received
0929 03:17:41.018032 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:41.018033 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:41.018044 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:41.018096 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:17:41.018220 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:41.018938 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:41.019665 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:41.019965 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:41.020037 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:41.020202 (+ 165us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:17:41.020233 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:17:41.020285 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0929 03:17:41.020335 (+ 50us) server_negotiation.cc:299] Negotiation successful
0929 03:17:41.020353 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260929 03:17:41.022944 23363 raft_consensus.cc:1261] T 16dcef6c559c40eaab699b371c1ae99c P f87970e42bbb4b10884b1606288565ca [term 1 FOLLOWER]: Refusing update from remote peer 7502a5d6be1646cba535265e54b2f07e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:17:41.023027 23223 raft_consensus.cc:1261] T 16dcef6c559c40eaab699b371c1ae99c P d68a40cd55a24edbb306ac354251f160 [term 1 FOLLOWER]: Refusing update from remote peer 7502a5d6be1646cba535265e54b2f07e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:17:41.023190 23443 consensus_queue.cc:1035] T 16dcef6c559c40eaab699b371c1ae99c P 7502a5d6be1646cba535265e54b2f07e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f87970e42bbb4b10884b1606288565ca" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 35231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:17:41.023303 23439 consensus_queue.cc:1035] T 16dcef6c559c40eaab699b371c1ae99c P 7502a5d6be1646cba535265e54b2f07e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d68a40cd55a24edbb306ac354251f160" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 34631 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:17:41.025285 23464 mvcc.cc:205] Tried to move back new op lower bound from 7334510022748008448 to 7334510021472968704. Current Snapshot: MvccSnapshot[applied={T|T < 7334510022748008448}]
I20260929 03:17:41.035849 23456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:41.033112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.66:59419 (local address 127.20.178.65:43333)
0929 03:17:41.033176 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:41.033178 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:41.033264 (+ 86us) server_negotiation.cc:407] Connection header received
0929 03:17:41.033318 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:41.033319 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:41.033327 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:41.033367 (+ 40us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:17:41.033497 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:41.034152 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:41.035245 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:41.035414 (+ 169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:41.035464 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:41.035649 (+ 185us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:17:41.035672 (+ 23us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:17:41.035708 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0929 03:17:41.035784 (+ 76us) server_negotiation.cc:299] Negotiation successful
0929 03:17:41.035800 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260929 03:17:41.035912 23480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:41.033027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.65:43333 (local address 127.20.178.66:59419)
0929 03:17:41.033186 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0929 03:17:41.033198 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 03:17:41.033261 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:17:41.033379 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:17:41.033380 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:17:41.033381 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 03:17:41.033446 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:41.033448 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:41.034223 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:41.034224 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:41.035154 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:41.035157 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:41.035484 (+ 327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:41.035485 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:41.035552 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:41.035566 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 03:17:41.035728 (+ 162us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 03:17:41.035730 (+ 2us) client_negotiation.cc:769] Sending connection context
0929 03:17:41.035783 (+ 53us) client_negotiation.cc:240] Negotiation successful
0929 03:17:41.035825 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":99,"threads_started":1}
I20260929 03:17:41.045156 22924 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37156:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:17:41.045286 22924 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37156:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:17:41.046021 22924 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 21c6f00f395f4ca0af3dc2144bf92b8f: Sending DeleteTablet for 3 replicas of tablet 16dcef6c559c40eaab699b371c1ae99c
I20260929 03:17:41.046320 23063 tablet_service.cc:1514] Processing DeleteTablet for tablet 16dcef6c559c40eaab699b371c1ae99c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:17:41 UTC) from {username='kudu', principal='kudu/127.20.178.126@KRBTEST.COM'} at 127.0.0.1:33500
I20260929 03:17:41.046429 23341 tablet_service.cc:1514] Processing DeleteTablet for tablet 16dcef6c559c40eaab699b371c1ae99c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:17:41 UTC) from {username='kudu', principal='kudu/127.20.178.126@KRBTEST.COM'} at 127.0.0.1:49072
I20260929 03:17:41.046653 23484 tablet_replica.cc:330] T 16dcef6c559c40eaab699b371c1ae99c P f87970e42bbb4b10884b1606288565ca: stopping tablet replica
I20260929 03:17:41.046754 23484 raft_consensus.cc:2229] T 16dcef6c559c40eaab699b371c1ae99c P f87970e42bbb4b10884b1606288565ca [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:17:41.046813 23484 raft_consensus.cc:2258] T 16dcef6c559c40eaab699b371c1ae99c P f87970e42bbb4b10884b1606288565ca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:17:41.047070 22924 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37156:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260929 03:17:41.047192 23484 ts_tablet_manager.cc:1905] T 16dcef6c559c40eaab699b371c1ae99c P f87970e42bbb4b10884b1606288565ca: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260929 03:17:41.047209 22924 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 03:17:41.047801 23203 tablet_service.cc:1514] Processing DeleteTablet for tablet 16dcef6c559c40eaab699b371c1ae99c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:17:41 UTC) from {username='kudu', principal='kudu/127.20.178.126@KRBTEST.COM'} at 127.0.0.1:38732
I20260929 03:17:41.048518 23063 tablet_service.cc:1467] Processing CreateTablet for tablet ac35041a0f6b4b55b02f9133e127b124 (DEFAULT_TABLE table=test-table [id=22f34635dfc8483b94f8871901a76942]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:17:41.048790 23063 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac35041a0f6b4b55b02f9133e127b124. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:17:41.049561 23484 ts_tablet_manager.cc:1918] T 16dcef6c559c40eaab699b371c1ae99c P f87970e42bbb4b10884b1606288565ca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 03:17:41.049625 23484 log.cc:1196] T 16dcef6c559c40eaab699b371c1ae99c P f87970e42bbb4b10884b1606288565ca: Deleting WAL directory at /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651852476208-21193-0/minicluster-data/ts-2/wal/wals/16dcef6c559c40eaab699b371c1ae99c
I20260929 03:17:41.049873 23484 ts_tablet_manager.cc:1939] T 16dcef6c559c40eaab699b371c1ae99c P f87970e42bbb4b10884b1606288565ca: Deleting consensus metadata
I20260929 03:17:41.050238 23433 tablet_bootstrap.cc:492] T ac35041a0f6b4b55b02f9133e127b124 P 7502a5d6be1646cba535265e54b2f07e: Bootstrap starting.
I20260929 03:17:41.050637 23433 tablet_bootstrap.cc:654] T ac35041a0f6b4b55b02f9133e127b124 P 7502a5d6be1646cba535265e54b2f07e: Neither blocks nor log segments found. Creating new log.
I20260929 03:17:41.051406 22912 catalog_manager.cc:4841] TS f87970e42bbb4b10884b1606288565ca (127.20.178.67:35231): tablet 16dcef6c559c40eaab699b371c1ae99c (table test-table [id=148299c5a13c4a75af4c315c0ee3916b]) successfully deleted
I20260929 03:17:41.051558 23341 tablet_service.cc:1467] Processing CreateTablet for tablet ac35041a0f6b4b55b02f9133e127b124 (DEFAULT_TABLE table=test-table [id=22f34635dfc8483b94f8871901a76942]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:17:41.051759 23341 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac35041a0f6b4b55b02f9133e127b124. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:17:41.052717 23435 tablet_bootstrap.cc:492] T ac35041a0f6b4b55b02f9133e127b124 P f87970e42bbb4b10884b1606288565ca: Bootstrap starting.
I20260929 03:17:41.053138 23435 tablet_bootstrap.cc:654] T ac35041a0f6b4b55b02f9133e127b124 P f87970e42bbb4b10884b1606288565ca: Neither blocks nor log segments found. Creating new log.
I20260929 03:17:41.053181 23433 tablet_bootstrap.cc:492] T ac35041a0f6b4b55b02f9133e127b124 P 7502a5d6be1646cba535265e54b2f07e: No bootstrap required, opened a new log
I20260929 03:17:41.053224 23433 ts_tablet_manager.cc:1397] T ac35041a0f6b4b55b02f9133e127b124 P 7502a5d6be1646cba535265e54b2f07e: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260929 03:17:41.053375 23433 raft_consensus.cc:348] T ac35041a0f6b4b55b02f9133e127b124 P 7502a5d6be1646cba535265e54b2f07e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7502a5d6be1646cba535265e54b2f07e" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 43333 } } peers { permanent_uuid: "d68a40cd55a24edbb306ac354251f160" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 34631 } } peers { permanent_uuid: "f87970e42bbb4b10884b1606288565ca" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 35231 } }
I20260929 03:17:41.053447 23433 raft_consensus.cc:374] T ac35041a0f6b4b55b02f9133e127b124 P 7502a5d6be1646cba535265e54b2f07e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:17:41.053473 23433 raft_consensus.cc:729] T ac35041a0f6b4b55b02f9133e127b124 P 7502a5d6be1646cba535265e54b2f07e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7502a5d6be1646cba535265e54b2f07e, State: Initialized, Role: FOLLOWER
I20260929 03:17:41.053515 23433 consensus_queue.cc:260] T ac35041a0f6b4b55b02f9133e127b124 P 7502a5d6be1646cba535265e54b2f07e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7502a5d6be1646cba535265e54b2f07e" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 43333 } } peers { permanent_uuid: "d68a40cd55a24edbb306ac354251f160" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 34631 } } peers { permanent_uuid: "f87970e42bbb4b10884b1606288565ca" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 35231 } }
I20260929 03:17:41.053618 23433 ts_tablet_manager.cc:1428] T ac35041a0f6b4b55b02f9133e127b124 P 7502a5d6be1646cba535265e54b2f07e: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260929 03:17:41.054010 23435 tablet_bootstrap.cc:492] T ac35041a0f6b4b55b02f9133e127b124 P f87970e42bbb4b10884b1606288565ca: No bootstrap required, opened a new log
I20260929 03:17:41.054061 23435 ts_tablet_manager.cc:1397] T ac35041a0f6b4b55b02f9133e127b124 P f87970e42bbb4b10884b1606288565ca: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260929 03:17:41.054200 23203 tablet_service.cc:1467] Processing CreateTablet for tablet ac35041a0f6b4b55b02f9133e127b124 (DEFAULT_TABLE table=test-table [id=22f34635dfc8483b94f8871901a76942]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:17:41.054194 23435 raft_consensus.cc:348] T ac35041a0f6b4b55b02f9133e127b124 P f87970e42bbb4b10884b1606288565ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7502a5d6be1646cba535265e54b2f07e" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 43333 } } peers { permanent_uuid: "d68a40cd55a24edbb306ac354251f160" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 34631 } } peers { permanent_uuid: "f87970e42bbb4b10884b1606288565ca" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 35231 } }
I20260929 03:17:41.054277 23435 raft_consensus.cc:374] T ac35041a0f6b4b55b02f9133e127b124 P f87970e42bbb4b10884b1606288565ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:17:41.054301 23435 raft_consensus.cc:729] T ac35041a0f6b4b55b02f9133e127b124 P f87970e42bbb4b10884b1606288565ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f87970e42bbb4b10884b1606288565ca, State: Initialized, Role: FOLLOWER
I20260929 03:17:41.054299 23203 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac35041a0f6b4b55b02f9133e127b124. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:17:41.054343 23435 consensus_queue.cc:260] T ac35041a0f6b4b55b02f9133e127b124 P f87970e42bbb4b10884b1606288565ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7502a5d6be1646cba535265e54b2f07e" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 43333 } } peers { permanent_uuid: "d68a40cd55a24edbb306ac354251f160" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 34631 } } peers { permanent_uuid: "f87970e42bbb4b10884b1606288565ca" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 35231 } }
I20260929 03:17:41.054437 23435 ts_tablet_manager.cc:1428] T ac35041a0f6b4b55b02f9133e127b124 P f87970e42bbb4b10884b1606288565ca: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260929 03:17:41.055370 23432 tablet_bootstrap.cc:492] T ac35041a0f6b4b55b02f9133e127b124 P d68a40cd55a24edbb306ac354251f160: Bootstrap starting.
I20260929 03:17:41.055776 23432 tablet_bootstrap.cc:654] T ac35041a0f6b4b55b02f9133e127b124 P d68a40cd55a24edbb306ac354251f160: Neither blocks nor log segments found. Creating new log.
I20260929 03:17:41.056082 23483 tablet_replica.cc:330] T 16dcef6c559c40eaab699b371c1ae99c P 7502a5d6be1646cba535265e54b2f07e: stopping tablet replica
I20260929 03:17:41.056186 23483 raft_consensus.cc:2229] T 16dcef6c559c40eaab699b371c1ae99c P 7502a5d6be1646cba535265e54b2f07e [term 1 LEADER]: Raft consensus shutting down.
I20260929 03:17:41.056300 23483 raft_consensus.cc:2258] T 16dcef6c559c40eaab699b371c1ae99c P 7502a5d6be1646cba535265e54b2f07e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:17:41.056658 23483 ts_tablet_manager.cc:1905] T 16dcef6c559c40eaab699b371c1ae99c P 7502a5d6be1646cba535265e54b2f07e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:17:41.056741 23432 tablet_bootstrap.cc:492] T ac35041a0f6b4b55b02f9133e127b124 P d68a40cd55a24edbb306ac354251f160: No bootstrap required, opened a new log
I20260929 03:17:41.056779 23432 ts_tablet_manager.cc:1397] T ac35041a0f6b4b55b02f9133e127b124 P d68a40cd55a24edbb306ac354251f160: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260929 03:17:41.057107 23432 raft_consensus.cc:348] T ac35041a0f6b4b55b02f9133e127b124 P d68a40cd55a24edbb306ac354251f160 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7502a5d6be1646cba535265e54b2f07e" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 43333 } } peers { permanent_uuid: "d68a40cd55a24edbb306ac354251f160" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 34631 } } peers { permanent_uuid: "f87970e42bbb4b10884b1606288565ca" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 35231 } }
I20260929 03:17:41.057174 23432 raft_consensus.cc:374] T ac35041a0f6b4b55b02f9133e127b124 P d68a40cd55a24edbb306ac354251f160 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:17:41.057195 23432 raft_consensus.cc:729] T ac35041a0f6b4b55b02f9133e127b124 P d68a40cd55a24edbb306ac354251f160 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d68a40cd55a24edbb306ac354251f160, State: Initialized, Role: FOLLOWER
I20260929 03:17:41.057247 23432 consensus_queue.cc:260] T ac35041a0f6b4b55b02f9133e127b124 P d68a40cd55a24edbb306ac354251f160 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7502a5d6be1646cba535265e54b2f07e" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 43333 } } peers { permanent_uuid: "d68a40cd55a24edbb306ac354251f160" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 34631 } } peers { permanent_uuid: "f87970e42bbb4b10884b1606288565ca" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 35231 } }
I20260929 03:17:41.057336 23432 ts_tablet_manager.cc:1428] T ac35041a0f6b4b55b02f9133e127b124 P d68a40cd55a24edbb306ac354251f160: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260929 03:17:41.058559 23483 ts_tablet_manager.cc:1918] T 16dcef6c559c40eaab699b371c1ae99c P 7502a5d6be1646cba535265e54b2f07e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 03:17:41.058650 23483 log.cc:1196] T 16dcef6c559c40eaab699b371c1ae99c P 7502a5d6be1646cba535265e54b2f07e: Deleting WAL directory at /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651852476208-21193-0/minicluster-data/ts-0/wal/wals/16dcef6c559c40eaab699b371c1ae99c
I20260929 03:17:41.058881 23483 ts_tablet_manager.cc:1939] T 16dcef6c559c40eaab699b371c1ae99c P 7502a5d6be1646cba535265e54b2f07e: Deleting consensus metadata
I20260929 03:17:41.059262 22913 catalog_manager.cc:4841] TS 7502a5d6be1646cba535265e54b2f07e (127.20.178.65:43333): tablet 16dcef6c559c40eaab699b371c1ae99c (table test-table [id=148299c5a13c4a75af4c315c0ee3916b]) successfully deleted
I20260929 03:17:41.060743 23485 tablet_replica.cc:330] T 16dcef6c559c40eaab699b371c1ae99c P d68a40cd55a24edbb306ac354251f160: stopping tablet replica
I20260929 03:17:41.060858 23485 raft_consensus.cc:2229] T 16dcef6c559c40eaab699b371c1ae99c P d68a40cd55a24edbb306ac354251f160 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:17:41.060918 23485 raft_consensus.cc:2258] T 16dcef6c559c40eaab699b371c1ae99c P d68a40cd55a24edbb306ac354251f160 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:17:41.061239 23485 ts_tablet_manager.cc:1905] T 16dcef6c559c40eaab699b371c1ae99c P d68a40cd55a24edbb306ac354251f160: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:17:41.062716 23485 ts_tablet_manager.cc:1918] T 16dcef6c559c40eaab699b371c1ae99c P d68a40cd55a24edbb306ac354251f160: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 03:17:41.062795 23485 log.cc:1196] T 16dcef6c559c40eaab699b371c1ae99c P d68a40cd55a24edbb306ac354251f160: Deleting WAL directory at /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651852476208-21193-0/minicluster-data/ts-1/wal/wals/16dcef6c559c40eaab699b371c1ae99c
I20260929 03:17:41.062793 23440 raft_consensus.cc:482] T ac35041a0f6b4b55b02f9133e127b124 P f87970e42bbb4b10884b1606288565ca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:17:41.062860 23440 raft_consensus.cc:504] T ac35041a0f6b4b55b02f9133e127b124 P f87970e42bbb4b10884b1606288565ca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7502a5d6be1646cba535265e54b2f07e" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 43333 } } peers { permanent_uuid: "d68a40cd55a24edbb306ac354251f160" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 34631 } } peers { permanent_uuid: "f87970e42bbb4b10884b1606288565ca" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 35231 } }
I20260929 03:17:41.063060 23485 ts_tablet_manager.cc:1939] T 16dcef6c559c40eaab699b371c1ae99c P d68a40cd55a24edbb306ac354251f160: Deleting consensus metadata
I20260929 03:17:41.063180 23440 leader_election.cc:288] T ac35041a0f6b4b55b02f9133e127b124 P f87970e42bbb4b10884b1606288565ca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7502a5d6be1646cba535265e54b2f07e (127.20.178.65:43333), d68a40cd55a24edbb306ac354251f160 (127.20.178.66:34631)
I20260929 03:17:41.063941 22910 catalog_manager.cc:4841] TS d68a40cd55a24edbb306ac354251f160 (127.20.178.66:34631): tablet 16dcef6c559c40eaab699b371c1ae99c (table test-table [id=148299c5a13c4a75af4c315c0ee3916b]) successfully deleted
I20260929 03:17:41.066501 23456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:41.063877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.67:60941 (local address 127.20.178.65:43333)
0929 03:17:41.063915 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:41.063917 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:41.063925 (+ 8us) server_negotiation.cc:407] Connection header received
0929 03:17:41.063947 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:41.063948 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:41.063956 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:41.064006 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:17:41.064290 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:41.064890 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:41.065818 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:41.066178 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:41.066263 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:41.066272 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0929 03:17:41.066412 (+ 140us) server_negotiation.cc:299] Negotiation successful
0929 03:17:41.066445 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260929 03:17:41.066522 23489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:41.063367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.65:43333 (local address 127.20.178.67:60941)
0929 03:17:41.063518 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0929 03:17:41.063530 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 03:17:41.063580 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:17:41.064056 (+ 476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:17:41.064057 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:17:41.064059 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:17:41.064135 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:41.064137 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:41.064968 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:41.064969 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:41.065723 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:41.065724 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:41.066262 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:41.066263 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:41.066330 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:41.066333 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 03:17:41.066393 (+ 60us) client_negotiation.cc:240] Negotiation successful
0929 03:17:41.066435 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":103,"threads_started":1}
I20260929 03:17:41.066771 23083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac35041a0f6b4b55b02f9133e127b124" candidate_uuid: "f87970e42bbb4b10884b1606288565ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7502a5d6be1646cba535265e54b2f07e" is_pre_election: true
I20260929 03:17:41.066851 23083 raft_consensus.cc:2454] T ac35041a0f6b4b55b02f9133e127b124 P 7502a5d6be1646cba535265e54b2f07e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f87970e42bbb4b10884b1606288565ca in term 0.
I20260929 03:17:41.067014 23297 leader_election.cc:302] T ac35041a0f6b4b55b02f9133e127b124 P f87970e42bbb4b10884b1606288565ca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7502a5d6be1646cba535265e54b2f07e, f87970e42bbb4b10884b1606288565ca; no voters:
I20260929 03:17:41.067164 23440 raft_consensus.cc:2781] T ac35041a0f6b4b55b02f9133e127b124 P f87970e42bbb4b10884b1606288565ca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 03:17:41.067217 23440 raft_consensus.cc:482] T ac35041a0f6b4b55b02f9133e127b124 P f87970e42bbb4b10884b1606288565ca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 03:17:41.067241 23440 raft_consensus.cc:3037] T ac35041a0f6b4b55b02f9133e127b124 P f87970e42bbb4b10884b1606288565ca [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:17:41.068074 23440 raft_consensus.cc:504] T ac35041a0f6b4b55b02f9133e127b124 P f87970e42bbb4b10884b1606288565ca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7502a5d6be1646cba535265e54b2f07e" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 43333 } } peers { permanent_uuid: "d68a40cd55a24edbb306ac354251f160" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 34631 } } peers { permanent_uuid: "f87970e42bbb4b10884b1606288565ca" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 35231 } }
I20260929 03:17:41.068235 23440 leader_election.cc:288] T ac35041a0f6b4b55b02f9133e127b124 P f87970e42bbb4b10884b1606288565ca [CANDIDATE]: Term 1 election: Requested vote from peers 7502a5d6be1646cba535265e54b2f07e (127.20.178.65:43333), d68a40cd55a24edbb306ac354251f160 (127.20.178.66:34631)
I20260929 03:17:41.068439 23083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac35041a0f6b4b55b02f9133e127b124" candidate_uuid: "f87970e42bbb4b10884b1606288565ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7502a5d6be1646cba535265e54b2f07e"
I20260929 03:17:41.068532 23083 raft_consensus.cc:3037] T ac35041a0f6b4b55b02f9133e127b124 P 7502a5d6be1646cba535265e54b2f07e [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:17:41.069339 23083 raft_consensus.cc:2454] T ac35041a0f6b4b55b02f9133e127b124 P 7502a5d6be1646cba535265e54b2f07e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f87970e42bbb4b10884b1606288565ca in term 1.
I20260929 03:17:41.069514 23297 leader_election.cc:302] T ac35041a0f6b4b55b02f9133e127b124 P f87970e42bbb4b10884b1606288565ca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7502a5d6be1646cba535265e54b2f07e, f87970e42bbb4b10884b1606288565ca; no voters:
I20260929 03:17:41.069588 23440 raft_consensus.cc:2781] T ac35041a0f6b4b55b02f9133e127b124 P f87970e42bbb4b10884b1606288565ca [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:17:41.069708 23440 raft_consensus.cc:686] T ac35041a0f6b4b55b02f9133e127b124 P f87970e42bbb4b10884b1606288565ca [term 1 LEADER]: Becoming Leader. State: Replica: f87970e42bbb4b10884b1606288565ca, State: Running, Role: LEADER
I20260929 03:17:41.069806 23440 consensus_queue.cc:237] T ac35041a0f6b4b55b02f9133e127b124 P f87970e42bbb4b10884b1606288565ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7502a5d6be1646cba535265e54b2f07e" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 43333 } } peers { permanent_uuid: "d68a40cd55a24edbb306ac354251f160" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 34631 } } peers { permanent_uuid: "f87970e42bbb4b10884b1606288565ca" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 35231 } }
I20260929 03:17:41.070386 22925 catalog_manager.cc:5495] T ac35041a0f6b4b55b02f9133e127b124 P f87970e42bbb4b10884b1606288565ca reported cstate change: term changed from 0 to 1, leader changed from <none> to f87970e42bbb4b10884b1606288565ca (127.20.178.67). New cstate: current_term: 1 leader_uuid: "f87970e42bbb4b10884b1606288565ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7502a5d6be1646cba535265e54b2f07e" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 43333 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d68a40cd55a24edbb306ac354251f160" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 34631 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f87970e42bbb4b10884b1606288565ca" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 35231 } health_report { overall_health: HEALTHY } } }
I20260929 03:17:41.071969 23431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:41.063853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.67:47691 (local address 127.20.178.66:34631)
0929 03:17:41.063907 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:41.063909 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:41.063919 (+ 10us) server_negotiation.cc:407] Connection header received
0929 03:17:41.063942 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:41.063943 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:41.063954 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:41.064020 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:17:41.068837 (+ 4817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:41.069521 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:41.070998 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:41.071543 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:41.071633 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:41.071643 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0929 03:17:41.071858 (+ 215us) server_negotiation.cc:299] Negotiation successful
0929 03:17:41.071894 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260929 03:17:41.072350 23490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:41.063600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.66:34631 (local address 127.20.178.67:47691)
0929 03:17:41.063710 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0929 03:17:41.063719 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 03:17:41.063760 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:17:41.068682 (+ 4922us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:17:41.068683 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:17:41.068686 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:17:41.068741 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:41.068743 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:41.069699 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:41.069699 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:41.070894 (+ 1195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:41.070897 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:41.071701 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:41.071701 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:41.071772 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:41.071776 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 03:17:41.072187 (+ 411us) client_negotiation.cc:240] Negotiation successful
0929 03:17:41.072256 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":79,"threads_started":1}
I20260929 03:17:41.072799 23223 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac35041a0f6b4b55b02f9133e127b124" candidate_uuid: "f87970e42bbb4b10884b1606288565ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d68a40cd55a24edbb306ac354251f160" is_pre_election: true
I20260929 03:17:41.072854 23222 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac35041a0f6b4b55b02f9133e127b124" candidate_uuid: "f87970e42bbb4b10884b1606288565ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d68a40cd55a24edbb306ac354251f160"
I20260929 03:17:41.072887 23223 raft_consensus.cc:2454] T ac35041a0f6b4b55b02f9133e127b124 P d68a40cd55a24edbb306ac354251f160 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f87970e42bbb4b10884b1606288565ca in term 0.
I20260929 03:17:41.077625 23430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:41.075167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49086 (local address 127.20.178.67:35231)
0929 03:17:41.075204 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:41.075206 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:41.075250 (+ 44us) server_negotiation.cc:407] Connection header received
0929 03:17:41.075334 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:41.075334 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:41.075353 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:41.075420 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:17:41.075517 (+ 97us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:41.075979 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:41.076787 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:41.077045 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:41.077127 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:41.077300 (+ 173us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:17:41.077406 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:17:41.077488 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 03:17:41.077528 (+ 40us) server_negotiation.cc:299] Negotiation successful
0929 03:17:41.077548 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260929 03:17:41.082206 23431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:41.079825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.67:42905 (local address 127.20.178.66:34631)
0929 03:17:41.079923 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:41.079925 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:41.079981 (+ 56us) server_negotiation.cc:407] Connection header received
0929 03:17:41.080043 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:41.080044 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:41.080053 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:41.080107 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:17:41.080226 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:41.080755 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:41.081462 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:41.081665 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:41.081749 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:41.081952 (+ 203us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:17:41.081982 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:17:41.082028 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0929 03:17:41.082125 (+ 97us) server_negotiation.cc:299] Negotiation successful
0929 03:17:41.082143 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260929 03:17:41.082450 23495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:41.079784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.66:34631 (local address 127.20.178.67:42905)
0929 03:17:41.079930 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0929 03:17:41.079940 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 03:17:41.079996 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:17:41.080110 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:17:41.080111 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:17:41.080112 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 03:17:41.080167 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:41.080168 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:41.080825 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:41.080826 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:41.081376 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:41.081377 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:41.081785 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:41.081786 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:41.081846 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:41.081858 (+ 12us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 03:17:41.082027 (+ 169us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 03:17:41.082029 (+ 2us) client_negotiation.cc:769] Sending connection context
0929 03:17:41.082119 (+ 90us) client_negotiation.cc:240] Negotiation successful
0929 03:17:41.082184 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":84,"threads_started":1}
I20260929 03:17:41.084559 23083 raft_consensus.cc:1261] T ac35041a0f6b4b55b02f9133e127b124 P 7502a5d6be1646cba535265e54b2f07e [term 1 FOLLOWER]: Refusing update from remote peer f87970e42bbb4b10884b1606288565ca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:17:41.084659 23222 raft_consensus.cc:3037] T ac35041a0f6b4b55b02f9133e127b124 P d68a40cd55a24edbb306ac354251f160 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:17:41.084790 23440 consensus_queue.cc:1035] T ac35041a0f6b4b55b02f9133e127b124 P f87970e42bbb4b10884b1606288565ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "7502a5d6be1646cba535265e54b2f07e" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 43333 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:17:41.085625 23222 raft_consensus.cc:1261] T ac35041a0f6b4b55b02f9133e127b124 P d68a40cd55a24edbb306ac354251f160 [term 1 FOLLOWER]: Refusing update from remote peer f87970e42bbb4b10884b1606288565ca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:17:41.085817 23440 consensus_queue.cc:1035] T ac35041a0f6b4b55b02f9133e127b124 P f87970e42bbb4b10884b1606288565ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "d68a40cd55a24edbb306ac354251f160" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 34631 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:17:41.093053 23430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:41.090623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.66:60081 (local address 127.20.178.67:35231)
0929 03:17:41.090672 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:41.090674 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:41.090686 (+ 12us) server_negotiation.cc:407] Connection header received
0929 03:17:41.090772 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:41.090773 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:41.090781 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:41.090829 (+ 48us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:17:41.090957 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:41.091532 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:41.092265 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:41.092548 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:41.092608 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:41.092810 (+ 202us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:17:41.092840 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:17:41.092884 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0929 03:17:41.092972 (+ 88us) server_negotiation.cc:299] Negotiation successful
0929 03:17:41.092990 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260929 03:17:41.093060 23480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:41.090544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.67:35231 (local address 127.20.178.66:60081)
0929 03:17:41.090653 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0929 03:17:41.090663 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 03:17:41.090701 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:17:41.090834 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:17:41.090835 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:17:41.090837 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 03:17:41.090877 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:41.090879 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:41.091619 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:41.091621 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:41.092159 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:41.092161 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:41.092658 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:41.092659 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:41.092715 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:41.092720 (+ 5us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 03:17:41.092905 (+ 185us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 03:17:41.092906 (+ 1us) client_negotiation.cc:769] Sending connection context
0929 03:17:41.092953 (+ 47us) client_negotiation.cc:240] Negotiation successful
0929 03:17:41.092996 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66}
W20260929 03:17:41.093582 23463 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260929 03:17:42.619661 22925 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37156:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:17:42.619758 22925 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37156:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:17:42.620558 22925 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 21c6f00f395f4ca0af3dc2144bf92b8f: Sending DeleteTablet for 3 replicas of tablet ac35041a0f6b4b55b02f9133e127b124
I20260929 03:17:42.620823 23063 tablet_service.cc:1514] Processing DeleteTablet for tablet ac35041a0f6b4b55b02f9133e127b124 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:17:42 UTC) from {username='kudu', principal='kudu/127.20.178.126@KRBTEST.COM'} at 127.0.0.1:33500
I20260929 03:17:42.620855 23203 tablet_service.cc:1514] Processing DeleteTablet for tablet ac35041a0f6b4b55b02f9133e127b124 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:17:42 UTC) from {username='kudu', principal='kudu/127.20.178.126@KRBTEST.COM'} at 127.0.0.1:38732
I20260929 03:17:42.621045 23522 tablet_replica.cc:330] T ac35041a0f6b4b55b02f9133e127b124 P 7502a5d6be1646cba535265e54b2f07e: stopping tablet replica
I20260929 03:17:42.621062 23523 tablet_replica.cc:330] T ac35041a0f6b4b55b02f9133e127b124 P d68a40cd55a24edbb306ac354251f160: stopping tablet replica
I20260929 03:17:42.621119 23522 raft_consensus.cc:2229] T ac35041a0f6b4b55b02f9133e127b124 P 7502a5d6be1646cba535265e54b2f07e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:17:42.621140 23341 tablet_service.cc:1514] Processing DeleteTablet for tablet ac35041a0f6b4b55b02f9133e127b124 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:17:42 UTC) from {username='kudu', principal='kudu/127.20.178.126@KRBTEST.COM'} at 127.0.0.1:49072
I20260929 03:17:42.621167 23522 raft_consensus.cc:2258] T ac35041a0f6b4b55b02f9133e127b124 P 7502a5d6be1646cba535265e54b2f07e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:17:42.621176 23523 raft_consensus.cc:2229] T ac35041a0f6b4b55b02f9133e127b124 P d68a40cd55a24edbb306ac354251f160 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:17:42.621227 23523 raft_consensus.cc:2258] T ac35041a0f6b4b55b02f9133e127b124 P d68a40cd55a24edbb306ac354251f160 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:17:42.621409 23522 ts_tablet_manager.cc:1905] T ac35041a0f6b4b55b02f9133e127b124 P 7502a5d6be1646cba535265e54b2f07e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:17:42.621472 23523 ts_tablet_manager.cc:1905] T ac35041a0f6b4b55b02f9133e127b124 P d68a40cd55a24edbb306ac354251f160: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:17:42.622749 23524 tablet_replica.cc:330] T ac35041a0f6b4b55b02f9133e127b124 P f87970e42bbb4b10884b1606288565ca: stopping tablet replica
I20260929 03:17:42.622822 23524 raft_consensus.cc:2229] T ac35041a0f6b4b55b02f9133e127b124 P f87970e42bbb4b10884b1606288565ca [term 1 LEADER]: Raft consensus shutting down.
I20260929 03:17:42.622907 23524 raft_consensus.cc:2258] T ac35041a0f6b4b55b02f9133e127b124 P f87970e42bbb4b10884b1606288565ca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:17:42.623121 23524 ts_tablet_manager.cc:1905] T ac35041a0f6b4b55b02f9133e127b124 P f87970e42bbb4b10884b1606288565ca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:17:42.623150 23522 ts_tablet_manager.cc:1918] T ac35041a0f6b4b55b02f9133e127b124 P 7502a5d6be1646cba535265e54b2f07e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 03:17:42.623236 23522 log.cc:1196] T ac35041a0f6b4b55b02f9133e127b124 P 7502a5d6be1646cba535265e54b2f07e: Deleting WAL directory at /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651852476208-21193-0/minicluster-data/ts-0/wal/wals/ac35041a0f6b4b55b02f9133e127b124
I20260929 03:17:42.623486 23522 ts_tablet_manager.cc:1939] T ac35041a0f6b4b55b02f9133e127b124 P 7502a5d6be1646cba535265e54b2f07e: Deleting consensus metadata
I20260929 03:17:42.623831 22913 catalog_manager.cc:4841] TS 7502a5d6be1646cba535265e54b2f07e (127.20.178.65:43333): tablet ac35041a0f6b4b55b02f9133e127b124 (table test-table [id=22f34635dfc8483b94f8871901a76942]) successfully deleted
I20260929 03:17:42.624351 23523 ts_tablet_manager.cc:1918] T ac35041a0f6b4b55b02f9133e127b124 P d68a40cd55a24edbb306ac354251f160: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 03:17:42.624432 23523 log.cc:1196] T ac35041a0f6b4b55b02f9133e127b124 P d68a40cd55a24edbb306ac354251f160: Deleting WAL directory at /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651852476208-21193-0/minicluster-data/ts-1/wal/wals/ac35041a0f6b4b55b02f9133e127b124
I20260929 03:17:42.624698 23523 ts_tablet_manager.cc:1939] T ac35041a0f6b4b55b02f9133e127b124 P d68a40cd55a24edbb306ac354251f160: Deleting consensus metadata
I20260929 03:17:42.624943 23524 ts_tablet_manager.cc:1918] T ac35041a0f6b4b55b02f9133e127b124 P f87970e42bbb4b10884b1606288565ca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 03:17:42.625007 23524 log.cc:1196] T ac35041a0f6b4b55b02f9133e127b124 P f87970e42bbb4b10884b1606288565ca: Deleting WAL directory at /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651852476208-21193-0/minicluster-data/ts-2/wal/wals/ac35041a0f6b4b55b02f9133e127b124
I20260929 03:17:42.625099 22910 catalog_manager.cc:4841] TS d68a40cd55a24edbb306ac354251f160 (127.20.178.66:34631): tablet ac35041a0f6b4b55b02f9133e127b124 (table test-table [id=22f34635dfc8483b94f8871901a76942]) successfully deleted
I20260929 03:17:42.625229 23524 ts_tablet_manager.cc:1939] T ac35041a0f6b4b55b02f9133e127b124 P f87970e42bbb4b10884b1606288565ca: Deleting consensus metadata
I20260929 03:17:42.625732 22912 catalog_manager.cc:4841] TS f87970e42bbb4b10884b1606288565ca (127.20.178.67:35231): tablet ac35041a0f6b4b55b02f9133e127b124 (table test-table [id=22f34635dfc8483b94f8871901a76942]) successfully deleted
Sep 29 03:17:42 dist-test-slave-c9qh krb5kdc[22881](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651862, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 03:17:42 dist-test-slave-c9qh krb5kdc[22881](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651862, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.20.178.126@KRBTEST.COM
I20260929 03:17:42.648666 23532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:42.637922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37170 (local address 127.20.178.126:40307)
0929 03:17:42.638087 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:42.638089 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:42.638097 (+ 8us) server_negotiation.cc:407] Connection header received
0929 03:17:42.638183 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:42.638184 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:42.638203 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:42.638280 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:17:42.638882 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:42.639403 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:42.640174 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:42.640377 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:42.640462 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:42.646147 (+ 5685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:17:42.646149 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:17:42.646150 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:17:42.646158 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:17:42.647442 (+ 1284us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:42.647801 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:42.647802 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:42.647802 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:42.647863 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:42.648083 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:42.648084 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:42.648084 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:42.648205 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:17:42.648322 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0929 03:17:42.648418 (+ 96us) server_negotiation.cc:299] Negotiation successful
0929 03:17:42.648507 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":93,"threads_started":1}
I20260929 03:17:42.650218 22925 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37170:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 03:17:42.650401 22925 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 03:17:42.651782 23063 tablet_service.cc:1467] Processing CreateTablet for tablet a19c65ef93c148f8a294393e53f37c33 (DEFAULT_TABLE table=test-table [id=30a39496b55041579209694b76551f1e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:17:42.651826 23203 tablet_service.cc:1467] Processing CreateTablet for tablet a19c65ef93c148f8a294393e53f37c33 (DEFAULT_TABLE table=test-table [id=30a39496b55041579209694b76551f1e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:17:42.651923 23063 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a19c65ef93c148f8a294393e53f37c33. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:17:42.651909 23341 tablet_service.cc:1467] Processing CreateTablet for tablet a19c65ef93c148f8a294393e53f37c33 (DEFAULT_TABLE table=test-table [id=30a39496b55041579209694b76551f1e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:17:42.651981 23203 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a19c65ef93c148f8a294393e53f37c33. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:17:42.652009 23341 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a19c65ef93c148f8a294393e53f37c33. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:17:42.653213 23533 tablet_bootstrap.cc:492] T a19c65ef93c148f8a294393e53f37c33 P 7502a5d6be1646cba535265e54b2f07e: Bootstrap starting.
I20260929 03:17:42.653426 23534 tablet_bootstrap.cc:492] T a19c65ef93c148f8a294393e53f37c33 P f87970e42bbb4b10884b1606288565ca: Bootstrap starting.
I20260929 03:17:42.653568 23535 tablet_bootstrap.cc:492] T a19c65ef93c148f8a294393e53f37c33 P d68a40cd55a24edbb306ac354251f160: Bootstrap starting.
I20260929 03:17:42.653640 23533 tablet_bootstrap.cc:654] T a19c65ef93c148f8a294393e53f37c33 P 7502a5d6be1646cba535265e54b2f07e: Neither blocks nor log segments found. Creating new log.
I20260929 03:17:42.653910 23534 tablet_bootstrap.cc:654] T a19c65ef93c148f8a294393e53f37c33 P f87970e42bbb4b10884b1606288565ca: Neither blocks nor log segments found. Creating new log.
I20260929 03:17:42.653963 23535 tablet_bootstrap.cc:654] T a19c65ef93c148f8a294393e53f37c33 P d68a40cd55a24edbb306ac354251f160: Neither blocks nor log segments found. Creating new log.
I20260929 03:17:42.654351 23533 tablet_bootstrap.cc:492] T a19c65ef93c148f8a294393e53f37c33 P 7502a5d6be1646cba535265e54b2f07e: No bootstrap required, opened a new log
I20260929 03:17:42.654420 23533 ts_tablet_manager.cc:1397] T a19c65ef93c148f8a294393e53f37c33 P 7502a5d6be1646cba535265e54b2f07e: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260929 03:17:42.654505 23535 tablet_bootstrap.cc:492] T a19c65ef93c148f8a294393e53f37c33 P d68a40cd55a24edbb306ac354251f160: No bootstrap required, opened a new log
I20260929 03:17:42.654554 23535 ts_tablet_manager.cc:1397] T a19c65ef93c148f8a294393e53f37c33 P d68a40cd55a24edbb306ac354251f160: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260929 03:17:42.654556 23533 raft_consensus.cc:348] T a19c65ef93c148f8a294393e53f37c33 P 7502a5d6be1646cba535265e54b2f07e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7502a5d6be1646cba535265e54b2f07e" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 43333 } } peers { permanent_uuid: "d68a40cd55a24edbb306ac354251f160" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 34631 } } peers { permanent_uuid: "f87970e42bbb4b10884b1606288565ca" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 35231 } }
I20260929 03:17:42.654623 23533 raft_consensus.cc:374] T a19c65ef93c148f8a294393e53f37c33 P 7502a5d6be1646cba535265e54b2f07e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:17:42.654650 23533 raft_consensus.cc:729] T a19c65ef93c148f8a294393e53f37c33 P 7502a5d6be1646cba535265e54b2f07e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7502a5d6be1646cba535265e54b2f07e, State: Initialized, Role: FOLLOWER
I20260929 03:17:42.654688 23535 raft_consensus.cc:348] T a19c65ef93c148f8a294393e53f37c33 P d68a40cd55a24edbb306ac354251f160 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7502a5d6be1646cba535265e54b2f07e" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 43333 } } peers { permanent_uuid: "d68a40cd55a24edbb306ac354251f160" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 34631 } } peers { permanent_uuid: "f87970e42bbb4b10884b1606288565ca" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 35231 } }
I20260929 03:17:42.654719 23533 consensus_queue.cc:260] T a19c65ef93c148f8a294393e53f37c33 P 7502a5d6be1646cba535265e54b2f07e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7502a5d6be1646cba535265e54b2f07e" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 43333 } } peers { permanent_uuid: "d68a40cd55a24edbb306ac354251f160" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 34631 } } peers { permanent_uuid: "f87970e42bbb4b10884b1606288565ca" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 35231 } }
I20260929 03:17:42.654783 23535 raft_consensus.cc:374] T a19c65ef93c148f8a294393e53f37c33 P d68a40cd55a24edbb306ac354251f160 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:17:42.654812 23535 raft_consensus.cc:729] T a19c65ef93c148f8a294393e53f37c33 P d68a40cd55a24edbb306ac354251f160 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d68a40cd55a24edbb306ac354251f160, State: Initialized, Role: FOLLOWER
I20260929 03:17:42.654887 23533 ts_tablet_manager.cc:1428] T a19c65ef93c148f8a294393e53f37c33 P 7502a5d6be1646cba535265e54b2f07e: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260929 03:17:42.654924 23535 consensus_queue.cc:260] T a19c65ef93c148f8a294393e53f37c33 P d68a40cd55a24edbb306ac354251f160 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7502a5d6be1646cba535265e54b2f07e" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 43333 } } peers { permanent_uuid: "d68a40cd55a24edbb306ac354251f160" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 34631 } } peers { permanent_uuid: "f87970e42bbb4b10884b1606288565ca" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 35231 } }
I20260929 03:17:42.655102 23535 ts_tablet_manager.cc:1428] T a19c65ef93c148f8a294393e53f37c33 P d68a40cd55a24edbb306ac354251f160: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260929 03:17:42.655406 23534 tablet_bootstrap.cc:492] T a19c65ef93c148f8a294393e53f37c33 P f87970e42bbb4b10884b1606288565ca: No bootstrap required, opened a new log
I20260929 03:17:42.655464 23534 ts_tablet_manager.cc:1397] T a19c65ef93c148f8a294393e53f37c33 P f87970e42bbb4b10884b1606288565ca: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 03:17:42.655592 23534 raft_consensus.cc:348] T a19c65ef93c148f8a294393e53f37c33 P f87970e42bbb4b10884b1606288565ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7502a5d6be1646cba535265e54b2f07e" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 43333 } } peers { permanent_uuid: "d68a40cd55a24edbb306ac354251f160" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 34631 } } peers { permanent_uuid: "f87970e42bbb4b10884b1606288565ca" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 35231 } }
I20260929 03:17:42.655668 23534 raft_consensus.cc:374] T a19c65ef93c148f8a294393e53f37c33 P f87970e42bbb4b10884b1606288565ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:17:42.655695 23534 raft_consensus.cc:729] T a19c65ef93c148f8a294393e53f37c33 P f87970e42bbb4b10884b1606288565ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f87970e42bbb4b10884b1606288565ca, State: Initialized, Role: FOLLOWER
I20260929 03:17:42.655751 23534 consensus_queue.cc:260] T a19c65ef93c148f8a294393e53f37c33 P f87970e42bbb4b10884b1606288565ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7502a5d6be1646cba535265e54b2f07e" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 43333 } } peers { permanent_uuid: "d68a40cd55a24edbb306ac354251f160" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 34631 } } peers { permanent_uuid: "f87970e42bbb4b10884b1606288565ca" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 35231 } }
I20260929 03:17:42.655851 23534 ts_tablet_manager.cc:1428] T a19c65ef93c148f8a294393e53f37c33 P f87970e42bbb4b10884b1606288565ca: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260929 03:17:42.749261 23539 raft_consensus.cc:482] T a19c65ef93c148f8a294393e53f37c33 P 7502a5d6be1646cba535265e54b2f07e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:17:42.749396 23539 raft_consensus.cc:504] T a19c65ef93c148f8a294393e53f37c33 P 7502a5d6be1646cba535265e54b2f07e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7502a5d6be1646cba535265e54b2f07e" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 43333 } } peers { permanent_uuid: "d68a40cd55a24edbb306ac354251f160" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 34631 } } peers { permanent_uuid: "f87970e42bbb4b10884b1606288565ca" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 35231 } }
I20260929 03:17:42.749697 23539 leader_election.cc:288] T a19c65ef93c148f8a294393e53f37c33 P 7502a5d6be1646cba535265e54b2f07e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d68a40cd55a24edbb306ac354251f160 (127.20.178.66:34631), f87970e42bbb4b10884b1606288565ca (127.20.178.67:35231)
I20260929 03:17:42.749913 23222 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a19c65ef93c148f8a294393e53f37c33" candidate_uuid: "7502a5d6be1646cba535265e54b2f07e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d68a40cd55a24edbb306ac354251f160" is_pre_election: true
I20260929 03:17:42.749926 23363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a19c65ef93c148f8a294393e53f37c33" candidate_uuid: "7502a5d6be1646cba535265e54b2f07e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f87970e42bbb4b10884b1606288565ca" is_pre_election: true
I20260929 03:17:42.750028 23222 raft_consensus.cc:2454] T a19c65ef93c148f8a294393e53f37c33 P d68a40cd55a24edbb306ac354251f160 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7502a5d6be1646cba535265e54b2f07e in term 0.
I20260929 03:17:42.750065 23363 raft_consensus.cc:2454] T a19c65ef93c148f8a294393e53f37c33 P f87970e42bbb4b10884b1606288565ca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7502a5d6be1646cba535265e54b2f07e in term 0.
I20260929 03:17:42.750231 23014 leader_election.cc:302] T a19c65ef93c148f8a294393e53f37c33 P 7502a5d6be1646cba535265e54b2f07e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7502a5d6be1646cba535265e54b2f07e, d68a40cd55a24edbb306ac354251f160; no voters:
I20260929 03:17:42.750351 23539 raft_consensus.cc:2781] T a19c65ef93c148f8a294393e53f37c33 P 7502a5d6be1646cba535265e54b2f07e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 03:17:42.750412 23539 raft_consensus.cc:482] T a19c65ef93c148f8a294393e53f37c33 P 7502a5d6be1646cba535265e54b2f07e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 03:17:42.750428 23539 raft_consensus.cc:3037] T a19c65ef93c148f8a294393e53f37c33 P 7502a5d6be1646cba535265e54b2f07e [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:17:42.751157 23539 raft_consensus.cc:504] T a19c65ef93c148f8a294393e53f37c33 P 7502a5d6be1646cba535265e54b2f07e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7502a5d6be1646cba535265e54b2f07e" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 43333 } } peers { permanent_uuid: "d68a40cd55a24edbb306ac354251f160" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 34631 } } peers { permanent_uuid: "f87970e42bbb4b10884b1606288565ca" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 35231 } }
I20260929 03:17:42.751312 23539 leader_election.cc:288] T a19c65ef93c148f8a294393e53f37c33 P 7502a5d6be1646cba535265e54b2f07e [CANDIDATE]: Term 1 election: Requested vote from peers d68a40cd55a24edbb306ac354251f160 (127.20.178.66:34631), f87970e42bbb4b10884b1606288565ca (127.20.178.67:35231)
I20260929 03:17:42.751497 23222 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a19c65ef93c148f8a294393e53f37c33" candidate_uuid: "7502a5d6be1646cba535265e54b2f07e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d68a40cd55a24edbb306ac354251f160"
I20260929 03:17:42.751521 23363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a19c65ef93c148f8a294393e53f37c33" candidate_uuid: "7502a5d6be1646cba535265e54b2f07e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f87970e42bbb4b10884b1606288565ca"
I20260929 03:17:42.751593 23363 raft_consensus.cc:3037] T a19c65ef93c148f8a294393e53f37c33 P f87970e42bbb4b10884b1606288565ca [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:17:42.751600 23222 raft_consensus.cc:3037] T a19c65ef93c148f8a294393e53f37c33 P d68a40cd55a24edbb306ac354251f160 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:17:42.752210 23363 raft_consensus.cc:2454] T a19c65ef93c148f8a294393e53f37c33 P f87970e42bbb4b10884b1606288565ca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7502a5d6be1646cba535265e54b2f07e in term 1.
I20260929 03:17:42.752331 23222 raft_consensus.cc:2454] T a19c65ef93c148f8a294393e53f37c33 P d68a40cd55a24edbb306ac354251f160 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7502a5d6be1646cba535265e54b2f07e in term 1.
I20260929 03:17:42.752410 23016 leader_election.cc:302] T a19c65ef93c148f8a294393e53f37c33 P 7502a5d6be1646cba535265e54b2f07e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7502a5d6be1646cba535265e54b2f07e, f87970e42bbb4b10884b1606288565ca; no voters:
I20260929 03:17:42.752540 23539 raft_consensus.cc:2781] T a19c65ef93c148f8a294393e53f37c33 P 7502a5d6be1646cba535265e54b2f07e [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:17:42.752768 23539 raft_consensus.cc:686] T a19c65ef93c148f8a294393e53f37c33 P 7502a5d6be1646cba535265e54b2f07e [term 1 LEADER]: Becoming Leader. State: Replica: 7502a5d6be1646cba535265e54b2f07e, State: Running, Role: LEADER
I20260929 03:17:42.752844 23539 consensus_queue.cc:237] T a19c65ef93c148f8a294393e53f37c33 P 7502a5d6be1646cba535265e54b2f07e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7502a5d6be1646cba535265e54b2f07e" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 43333 } } peers { permanent_uuid: "d68a40cd55a24edbb306ac354251f160" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 34631 } } peers { permanent_uuid: "f87970e42bbb4b10884b1606288565ca" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 35231 } }
I20260929 03:17:42.753650 22925 catalog_manager.cc:5495] T a19c65ef93c148f8a294393e53f37c33 P 7502a5d6be1646cba535265e54b2f07e reported cstate change: term changed from 0 to 1, leader changed from <none> to 7502a5d6be1646cba535265e54b2f07e (127.20.178.65). New cstate: current_term: 1 leader_uuid: "7502a5d6be1646cba535265e54b2f07e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7502a5d6be1646cba535265e54b2f07e" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 43333 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d68a40cd55a24edbb306ac354251f160" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 34631 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f87970e42bbb4b10884b1606288565ca" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 35231 } health_report { overall_health: UNKNOWN } } }
I20260929 03:17:42.773797 23543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:42.771546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33532 (local address 127.20.178.65:43333)
0929 03:17:42.771714 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:42.771715 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:42.771722 (+ 7us) server_negotiation.cc:407] Connection header received
0929 03:17:42.771749 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:42.771750 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:42.771759 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:42.771816 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:17:42.771933 (+ 117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:42.772407 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:42.773203 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:42.773390 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:42.773442 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:42.773558 (+ 116us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:17:42.773583 (+ 25us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:17:42.773615 (+ 32us) server_negotiation.cc:1035] Waiting for connection context
0929 03:17:42.773673 (+ 58us) server_negotiation.cc:299] Negotiation successful
0929 03:17:42.773688 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":102,"threads_started":1}
I20260929 03:17:42.776175 23222 raft_consensus.cc:1261] T a19c65ef93c148f8a294393e53f37c33 P d68a40cd55a24edbb306ac354251f160 [term 1 FOLLOWER]: Refusing update from remote peer 7502a5d6be1646cba535265e54b2f07e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:17:42.776162 23363 raft_consensus.cc:1261] T a19c65ef93c148f8a294393e53f37c33 P f87970e42bbb4b10884b1606288565ca [term 1 FOLLOWER]: Refusing update from remote peer 7502a5d6be1646cba535265e54b2f07e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:17:42.776387 23539 consensus_queue.cc:1035] T a19c65ef93c148f8a294393e53f37c33 P 7502a5d6be1646cba535265e54b2f07e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f87970e42bbb4b10884b1606288565ca" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 35231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:17:42.776479 23540 consensus_queue.cc:1035] T a19c65ef93c148f8a294393e53f37c33 P 7502a5d6be1646cba535265e54b2f07e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d68a40cd55a24edbb306ac354251f160" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 34631 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:17:42.786809 22924 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37170:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:17:42.786901 22924 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37170:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:17:42.787513 22924 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 21c6f00f395f4ca0af3dc2144bf92b8f: Sending DeleteTablet for 3 replicas of tablet a19c65ef93c148f8a294393e53f37c33
I20260929 03:17:42.787894 23203 tablet_service.cc:1514] Processing DeleteTablet for tablet a19c65ef93c148f8a294393e53f37c33 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:17:42 UTC) from {username='kudu', principal='kudu/127.20.178.126@KRBTEST.COM'} at 127.0.0.1:38732
I20260929 03:17:42.787987 23523 tablet_replica.cc:330] T a19c65ef93c148f8a294393e53f37c33 P d68a40cd55a24edbb306ac354251f160: stopping tablet replica
I20260929 03:17:42.787954 23341 tablet_service.cc:1514] Processing DeleteTablet for tablet a19c65ef93c148f8a294393e53f37c33 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:17:42 UTC) from {username='kudu', principal='kudu/127.20.178.126@KRBTEST.COM'} at 127.0.0.1:49072
I20260929 03:17:42.788048 23523 raft_consensus.cc:2229] T a19c65ef93c148f8a294393e53f37c33 P d68a40cd55a24edbb306ac354251f160 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:17:42.788062 23524 tablet_replica.cc:330] T a19c65ef93c148f8a294393e53f37c33 P f87970e42bbb4b10884b1606288565ca: stopping tablet replica
I20260929 03:17:42.788097 23523 raft_consensus.cc:2258] T a19c65ef93c148f8a294393e53f37c33 P d68a40cd55a24edbb306ac354251f160 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:17:42.788111 23524 raft_consensus.cc:2229] T a19c65ef93c148f8a294393e53f37c33 P f87970e42bbb4b10884b1606288565ca [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:17:42.788154 23524 raft_consensus.cc:2258] T a19c65ef93c148f8a294393e53f37c33 P f87970e42bbb4b10884b1606288565ca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:17:42.788300 23063 tablet_service.cc:1514] Processing DeleteTablet for tablet a19c65ef93c148f8a294393e53f37c33 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:17:42 UTC) from {username='kudu', principal='kudu/127.20.178.126@KRBTEST.COM'} at 127.0.0.1:33500
I20260929 03:17:42.788422 23523 ts_tablet_manager.cc:1905] T a19c65ef93c148f8a294393e53f37c33 P d68a40cd55a24edbb306ac354251f160: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:17:42.788430 23522 tablet_replica.cc:330] T a19c65ef93c148f8a294393e53f37c33 P 7502a5d6be1646cba535265e54b2f07e: stopping tablet replica
I20260929 03:17:42.788493 22924 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37170:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260929 03:17:42.788555 23522 raft_consensus.cc:2229] T a19c65ef93c148f8a294393e53f37c33 P 7502a5d6be1646cba535265e54b2f07e [term 1 LEADER]: Raft consensus shutting down.
I20260929 03:17:42.788667 23522 raft_consensus.cc:2258] T a19c65ef93c148f8a294393e53f37c33 P 7502a5d6be1646cba535265e54b2f07e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:17:42.788807 23524 ts_tablet_manager.cc:1905] T a19c65ef93c148f8a294393e53f37c33 P f87970e42bbb4b10884b1606288565ca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:17:42.788986 23522 ts_tablet_manager.cc:1905] T a19c65ef93c148f8a294393e53f37c33 P 7502a5d6be1646cba535265e54b2f07e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:17:42.790251 23523 ts_tablet_manager.cc:1918] T a19c65ef93c148f8a294393e53f37c33 P d68a40cd55a24edbb306ac354251f160: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 03:17:42.790325 23523 log.cc:1196] T a19c65ef93c148f8a294393e53f37c33 P d68a40cd55a24edbb306ac354251f160: Deleting WAL directory at /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651852476208-21193-0/minicluster-data/ts-1/wal/wals/a19c65ef93c148f8a294393e53f37c33
I20260929 03:17:42.790551 23523 ts_tablet_manager.cc:1939] T a19c65ef93c148f8a294393e53f37c33 P d68a40cd55a24edbb306ac354251f160: Deleting consensus metadata
I20260929 03:17:42.790548 23524 ts_tablet_manager.cc:1918] T a19c65ef93c148f8a294393e53f37c33 P f87970e42bbb4b10884b1606288565ca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 03:17:42.790614 23524 log.cc:1196] T a19c65ef93c148f8a294393e53f37c33 P f87970e42bbb4b10884b1606288565ca: Deleting WAL directory at /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651852476208-21193-0/minicluster-data/ts-2/wal/wals/a19c65ef93c148f8a294393e53f37c33
I20260929 03:17:42.790742 23522 ts_tablet_manager.cc:1918] T a19c65ef93c148f8a294393e53f37c33 P 7502a5d6be1646cba535265e54b2f07e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 03:17:42.790802 23522 log.cc:1196] T a19c65ef93c148f8a294393e53f37c33 P 7502a5d6be1646cba535265e54b2f07e: Deleting WAL directory at /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651852476208-21193-0/minicluster-data/ts-0/wal/wals/a19c65ef93c148f8a294393e53f37c33
I20260929 03:17:42.790860 23524 ts_tablet_manager.cc:1939] T a19c65ef93c148f8a294393e53f37c33 P f87970e42bbb4b10884b1606288565ca: Deleting consensus metadata
I20260929 03:17:42.790879 22910 catalog_manager.cc:4841] TS d68a40cd55a24edbb306ac354251f160 (127.20.178.66:34631): tablet a19c65ef93c148f8a294393e53f37c33 (table test-table [id=30a39496b55041579209694b76551f1e]) successfully deleted
I20260929 03:17:42.791021 23522 ts_tablet_manager.cc:1939] T a19c65ef93c148f8a294393e53f37c33 P 7502a5d6be1646cba535265e54b2f07e: Deleting consensus metadata
W20260929 03:17:42.791018 22924 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 03:17:42.791380 22913 catalog_manager.cc:4841] TS 7502a5d6be1646cba535265e54b2f07e (127.20.178.65:43333): tablet a19c65ef93c148f8a294393e53f37c33 (table test-table [id=30a39496b55041579209694b76551f1e]) successfully deleted
I20260929 03:17:42.791426 22912 catalog_manager.cc:4841] TS f87970e42bbb4b10884b1606288565ca (127.20.178.67:35231): tablet a19c65ef93c148f8a294393e53f37c33 (table test-table [id=30a39496b55041579209694b76551f1e]) successfully deleted
I20260929 03:17:42.792280 23063 tablet_service.cc:1467] Processing CreateTablet for tablet f1a904cb2fed48e3a1e60f1dca6d7537 (DEFAULT_TABLE table=test-table [id=b8723c594032412a97d2ccfc2ee4401f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:17:42.792290 23203 tablet_service.cc:1467] Processing CreateTablet for tablet f1a904cb2fed48e3a1e60f1dca6d7537 (DEFAULT_TABLE table=test-table [id=b8723c594032412a97d2ccfc2ee4401f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:17:42.792304 23341 tablet_service.cc:1467] Processing CreateTablet for tablet f1a904cb2fed48e3a1e60f1dca6d7537 (DEFAULT_TABLE table=test-table [id=b8723c594032412a97d2ccfc2ee4401f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:17:42.792419 23203 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1a904cb2fed48e3a1e60f1dca6d7537. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:17:42.792419 23341 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1a904cb2fed48e3a1e60f1dca6d7537. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:17:42.792425 23063 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1a904cb2fed48e3a1e60f1dca6d7537. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:17:42.793463 23533 tablet_bootstrap.cc:492] T f1a904cb2fed48e3a1e60f1dca6d7537 P 7502a5d6be1646cba535265e54b2f07e: Bootstrap starting.
I20260929 03:17:42.793602 23535 tablet_bootstrap.cc:492] T f1a904cb2fed48e3a1e60f1dca6d7537 P d68a40cd55a24edbb306ac354251f160: Bootstrap starting.
I20260929 03:17:42.793678 23534 tablet_bootstrap.cc:492] T f1a904cb2fed48e3a1e60f1dca6d7537 P f87970e42bbb4b10884b1606288565ca: Bootstrap starting.
I20260929 03:17:42.793862 23533 tablet_bootstrap.cc:654] T f1a904cb2fed48e3a1e60f1dca6d7537 P 7502a5d6be1646cba535265e54b2f07e: Neither blocks nor log segments found. Creating new log.
I20260929 03:17:42.793962 23534 tablet_bootstrap.cc:654] T f1a904cb2fed48e3a1e60f1dca6d7537 P f87970e42bbb4b10884b1606288565ca: Neither blocks nor log segments found. Creating new log.
I20260929 03:17:42.793965 23535 tablet_bootstrap.cc:654] T f1a904cb2fed48e3a1e60f1dca6d7537 P d68a40cd55a24edbb306ac354251f160: Neither blocks nor log segments found. Creating new log.
I20260929 03:17:42.794536 23533 tablet_bootstrap.cc:492] T f1a904cb2fed48e3a1e60f1dca6d7537 P 7502a5d6be1646cba535265e54b2f07e: No bootstrap required, opened a new log
I20260929 03:17:42.794606 23533 ts_tablet_manager.cc:1397] T f1a904cb2fed48e3a1e60f1dca6d7537 P 7502a5d6be1646cba535265e54b2f07e: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260929 03:17:42.794620 23535 tablet_bootstrap.cc:492] T f1a904cb2fed48e3a1e60f1dca6d7537 P d68a40cd55a24edbb306ac354251f160: No bootstrap required, opened a new log
I20260929 03:17:42.794637 23534 tablet_bootstrap.cc:492] T f1a904cb2fed48e3a1e60f1dca6d7537 P f87970e42bbb4b10884b1606288565ca: No bootstrap required, opened a new log
I20260929 03:17:42.794677 23534 ts_tablet_manager.cc:1397] T f1a904cb2fed48e3a1e60f1dca6d7537 P f87970e42bbb4b10884b1606288565ca: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260929 03:17:42.794675 23535 ts_tablet_manager.cc:1397] T f1a904cb2fed48e3a1e60f1dca6d7537 P d68a40cd55a24edbb306ac354251f160: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260929 03:17:42.794718 23533 raft_consensus.cc:348] T f1a904cb2fed48e3a1e60f1dca6d7537 P 7502a5d6be1646cba535265e54b2f07e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7502a5d6be1646cba535265e54b2f07e" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 43333 } } peers { permanent_uuid: "d68a40cd55a24edbb306ac354251f160" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 34631 } } peers { permanent_uuid: "f87970e42bbb4b10884b1606288565ca" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 35231 } }
I20260929 03:17:42.794792 23533 raft_consensus.cc:374] T f1a904cb2fed48e3a1e60f1dca6d7537 P 7502a5d6be1646cba535265e54b2f07e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:17:42.794821 23533 raft_consensus.cc:729] T f1a904cb2fed48e3a1e60f1dca6d7537 P 7502a5d6be1646cba535265e54b2f07e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7502a5d6be1646cba535265e54b2f07e, State: Initialized, Role: FOLLOWER
I20260929 03:17:42.794788 23534 raft_consensus.cc:348] T f1a904cb2fed48e3a1e60f1dca6d7537 P f87970e42bbb4b10884b1606288565ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7502a5d6be1646cba535265e54b2f07e" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 43333 } } peers { permanent_uuid: "d68a40cd55a24edbb306ac354251f160" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 34631 } } peers { permanent_uuid: "f87970e42bbb4b10884b1606288565ca" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 35231 } }
I20260929 03:17:42.794790 23535 raft_consensus.cc:348] T f1a904cb2fed48e3a1e60f1dca6d7537 P d68a40cd55a24edbb306ac354251f160 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7502a5d6be1646cba535265e54b2f07e" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 43333 } } peers { permanent_uuid: "d68a40cd55a24edbb306ac354251f160" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 34631 } } peers { permanent_uuid: "f87970e42bbb4b10884b1606288565ca" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 35231 } }
I20260929 03:17:42.794852 23534 raft_consensus.cc:374] T f1a904cb2fed48e3a1e60f1dca6d7537 P f87970e42bbb4b10884b1606288565ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:17:42.794873 23534 raft_consensus.cc:729] T f1a904cb2fed48e3a1e60f1dca6d7537 P f87970e42bbb4b10884b1606288565ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f87970e42bbb4b10884b1606288565ca, State: Initialized, Role: FOLLOWER
I20260929 03:17:42.794873 23535 raft_consensus.cc:374] T f1a904cb2fed48e3a1e60f1dca6d7537 P d68a40cd55a24edbb306ac354251f160 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:17:42.794862 23533 consensus_queue.cc:260] T f1a904cb2fed48e3a1e60f1dca6d7537 P 7502a5d6be1646cba535265e54b2f07e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7502a5d6be1646cba535265e54b2f07e" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 43333 } } peers { permanent_uuid: "d68a40cd55a24edbb306ac354251f160" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 34631 } } peers { permanent_uuid: "f87970e42bbb4b10884b1606288565ca" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 35231 } }
I20260929 03:17:42.794903 23535 raft_consensus.cc:729] T f1a904cb2fed48e3a1e60f1dca6d7537 P d68a40cd55a24edbb306ac354251f160 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d68a40cd55a24edbb306ac354251f160, State: Initialized, Role: FOLLOWER
I20260929 03:17:42.794914 23534 consensus_queue.cc:260] T f1a904cb2fed48e3a1e60f1dca6d7537 P f87970e42bbb4b10884b1606288565ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7502a5d6be1646cba535265e54b2f07e" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 43333 } } peers { permanent_uuid: "d68a40cd55a24edbb306ac354251f160" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 34631 } } peers { permanent_uuid: "f87970e42bbb4b10884b1606288565ca" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 35231 } }
I20260929 03:17:42.794955 23533 ts_tablet_manager.cc:1428] T f1a904cb2fed48e3a1e60f1dca6d7537 P 7502a5d6be1646cba535265e54b2f07e: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260929 03:17:42.794953 23535 consensus_queue.cc:260] T f1a904cb2fed48e3a1e60f1dca6d7537 P d68a40cd55a24edbb306ac354251f160 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7502a5d6be1646cba535265e54b2f07e" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 43333 } } peers { permanent_uuid: "d68a40cd55a24edbb306ac354251f160" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 34631 } } peers { permanent_uuid: "f87970e42bbb4b10884b1606288565ca" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 35231 } }
I20260929 03:17:42.794993 23534 ts_tablet_manager.cc:1428] T f1a904cb2fed48e3a1e60f1dca6d7537 P f87970e42bbb4b10884b1606288565ca: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260929 03:17:42.795053 23535 ts_tablet_manager.cc:1428] T f1a904cb2fed48e3a1e60f1dca6d7537 P d68a40cd55a24edbb306ac354251f160: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260929 03:17:42.805259 23539 raft_consensus.cc:482] T f1a904cb2fed48e3a1e60f1dca6d7537 P 7502a5d6be1646cba535265e54b2f07e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:17:42.805346 23539 raft_consensus.cc:504] T f1a904cb2fed48e3a1e60f1dca6d7537 P 7502a5d6be1646cba535265e54b2f07e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7502a5d6be1646cba535265e54b2f07e" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 43333 } } peers { permanent_uuid: "d68a40cd55a24edbb306ac354251f160" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 34631 } } peers { permanent_uuid: "f87970e42bbb4b10884b1606288565ca" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 35231 } }
I20260929 03:17:42.805514 23539 leader_election.cc:288] T f1a904cb2fed48e3a1e60f1dca6d7537 P 7502a5d6be1646cba535265e54b2f07e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d68a40cd55a24edbb306ac354251f160 (127.20.178.66:34631), f87970e42bbb4b10884b1606288565ca (127.20.178.67:35231)
I20260929 03:17:42.805676 23363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1a904cb2fed48e3a1e60f1dca6d7537" candidate_uuid: "7502a5d6be1646cba535265e54b2f07e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f87970e42bbb4b10884b1606288565ca" is_pre_election: true
I20260929 03:17:42.805780 23363 raft_consensus.cc:2454] T f1a904cb2fed48e3a1e60f1dca6d7537 P f87970e42bbb4b10884b1606288565ca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7502a5d6be1646cba535265e54b2f07e in term 0.
I20260929 03:17:42.805718 23222 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1a904cb2fed48e3a1e60f1dca6d7537" candidate_uuid: "7502a5d6be1646cba535265e54b2f07e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d68a40cd55a24edbb306ac354251f160" is_pre_election: true
I20260929 03:17:42.805892 23222 raft_consensus.cc:2454] T f1a904cb2fed48e3a1e60f1dca6d7537 P d68a40cd55a24edbb306ac354251f160 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7502a5d6be1646cba535265e54b2f07e in term 0.
I20260929 03:17:42.805960 23016 leader_election.cc:302] T f1a904cb2fed48e3a1e60f1dca6d7537 P 7502a5d6be1646cba535265e54b2f07e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7502a5d6be1646cba535265e54b2f07e, f87970e42bbb4b10884b1606288565ca; no voters:
I20260929 03:17:42.806074 23539 raft_consensus.cc:2781] T f1a904cb2fed48e3a1e60f1dca6d7537 P 7502a5d6be1646cba535265e54b2f07e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 03:17:42.806118 23539 raft_consensus.cc:482] T f1a904cb2fed48e3a1e60f1dca6d7537 P 7502a5d6be1646cba535265e54b2f07e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 03:17:42.806143 23539 raft_consensus.cc:3037] T f1a904cb2fed48e3a1e60f1dca6d7537 P 7502a5d6be1646cba535265e54b2f07e [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:17:42.807004 23539 raft_consensus.cc:504] T f1a904cb2fed48e3a1e60f1dca6d7537 P 7502a5d6be1646cba535265e54b2f07e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7502a5d6be1646cba535265e54b2f07e" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 43333 } } peers { permanent_uuid: "d68a40cd55a24edbb306ac354251f160" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 34631 } } peers { permanent_uuid: "f87970e42bbb4b10884b1606288565ca" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 35231 } }
I20260929 03:17:42.807174 23539 leader_election.cc:288] T f1a904cb2fed48e3a1e60f1dca6d7537 P 7502a5d6be1646cba535265e54b2f07e [CANDIDATE]: Term 1 election: Requested vote from peers d68a40cd55a24edbb306ac354251f160 (127.20.178.66:34631), f87970e42bbb4b10884b1606288565ca (127.20.178.67:35231)
I20260929 03:17:42.807286 23222 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1a904cb2fed48e3a1e60f1dca6d7537" candidate_uuid: "7502a5d6be1646cba535265e54b2f07e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d68a40cd55a24edbb306ac354251f160"
I20260929 03:17:42.807379 23222 raft_consensus.cc:3037] T f1a904cb2fed48e3a1e60f1dca6d7537 P d68a40cd55a24edbb306ac354251f160 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:17:42.807328 23363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1a904cb2fed48e3a1e60f1dca6d7537" candidate_uuid: "7502a5d6be1646cba535265e54b2f07e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f87970e42bbb4b10884b1606288565ca"
I20260929 03:17:42.807426 23363 raft_consensus.cc:3037] T f1a904cb2fed48e3a1e60f1dca6d7537 P f87970e42bbb4b10884b1606288565ca [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:17:42.808005 23222 raft_consensus.cc:2454] T f1a904cb2fed48e3a1e60f1dca6d7537 P d68a40cd55a24edbb306ac354251f160 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7502a5d6be1646cba535265e54b2f07e in term 1.
I20260929 03:17:42.808018 23363 raft_consensus.cc:2454] T f1a904cb2fed48e3a1e60f1dca6d7537 P f87970e42bbb4b10884b1606288565ca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7502a5d6be1646cba535265e54b2f07e in term 1.
I20260929 03:17:42.808173 23014 leader_election.cc:302] T f1a904cb2fed48e3a1e60f1dca6d7537 P 7502a5d6be1646cba535265e54b2f07e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7502a5d6be1646cba535265e54b2f07e, d68a40cd55a24edbb306ac354251f160; no voters:
I20260929 03:17:42.808277 23539 raft_consensus.cc:2781] T f1a904cb2fed48e3a1e60f1dca6d7537 P 7502a5d6be1646cba535265e54b2f07e [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:17:42.808331 23539 raft_consensus.cc:686] T f1a904cb2fed48e3a1e60f1dca6d7537 P 7502a5d6be1646cba535265e54b2f07e [term 1 LEADER]: Becoming Leader. State: Replica: 7502a5d6be1646cba535265e54b2f07e, State: Running, Role: LEADER
I20260929 03:17:42.808382 23539 consensus_queue.cc:237] T f1a904cb2fed48e3a1e60f1dca6d7537 P 7502a5d6be1646cba535265e54b2f07e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7502a5d6be1646cba535265e54b2f07e" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 43333 } } peers { permanent_uuid: "d68a40cd55a24edbb306ac354251f160" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 34631 } } peers { permanent_uuid: "f87970e42bbb4b10884b1606288565ca" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 35231 } }
I20260929 03:17:42.809023 22926 catalog_manager.cc:5495] T f1a904cb2fed48e3a1e60f1dca6d7537 P 7502a5d6be1646cba535265e54b2f07e reported cstate change: term changed from 0 to 1, leader changed from <none> to 7502a5d6be1646cba535265e54b2f07e (127.20.178.65). New cstate: current_term: 1 leader_uuid: "7502a5d6be1646cba535265e54b2f07e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7502a5d6be1646cba535265e54b2f07e" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 43333 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d68a40cd55a24edbb306ac354251f160" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 34631 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f87970e42bbb4b10884b1606288565ca" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 35231 } health_report { overall_health: UNKNOWN } } }
I20260929 03:17:42.815094 23222 raft_consensus.cc:1261] T f1a904cb2fed48e3a1e60f1dca6d7537 P d68a40cd55a24edbb306ac354251f160 [term 1 FOLLOWER]: Refusing update from remote peer 7502a5d6be1646cba535265e54b2f07e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:17:42.815104 23363 raft_consensus.cc:1261] T f1a904cb2fed48e3a1e60f1dca6d7537 P f87970e42bbb4b10884b1606288565ca [term 1 FOLLOWER]: Refusing update from remote peer 7502a5d6be1646cba535265e54b2f07e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:17:42.815295 23539 consensus_queue.cc:1035] T f1a904cb2fed48e3a1e60f1dca6d7537 P 7502a5d6be1646cba535265e54b2f07e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d68a40cd55a24edbb306ac354251f160" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 34631 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:17:42.815376 23540 consensus_queue.cc:1035] T f1a904cb2fed48e3a1e60f1dca6d7537 P 7502a5d6be1646cba535265e54b2f07e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f87970e42bbb4b10884b1606288565ca" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 35231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260929 03:17:42.819485 23506 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260929 03:17:44.335193 22924 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37170:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:17:44.335314 22924 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37170:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:17:44.336090 22924 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 21c6f00f395f4ca0af3dc2144bf92b8f: Sending DeleteTablet for 3 replicas of tablet f1a904cb2fed48e3a1e60f1dca6d7537
I20260929 03:17:44.336398 23063 tablet_service.cc:1514] Processing DeleteTablet for tablet f1a904cb2fed48e3a1e60f1dca6d7537 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:17:44 UTC) from {username='kudu', principal='kudu/127.20.178.126@KRBTEST.COM'} at 127.0.0.1:33500
I20260929 03:17:44.336409 23341 tablet_service.cc:1514] Processing DeleteTablet for tablet f1a904cb2fed48e3a1e60f1dca6d7537 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:17:44 UTC) from {username='kudu', principal='kudu/127.20.178.126@KRBTEST.COM'} at 127.0.0.1:49072
I20260929 03:17:44.336545 21193 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuoaHbX/build/release/bin/kudu with pid 22998
I20260929 03:17:44.336596 23203 tablet_service.cc:1514] Processing DeleteTablet for tablet f1a904cb2fed48e3a1e60f1dca6d7537 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:17:44 UTC) from {username='kudu', principal='kudu/127.20.178.126@KRBTEST.COM'} at 127.0.0.1:38732
I20260929 03:17:44.339238 23577 tablet_replica.cc:330] T f1a904cb2fed48e3a1e60f1dca6d7537 P d68a40cd55a24edbb306ac354251f160: stopping tablet replica
I20260929 03:17:44.339303 23577 raft_consensus.cc:2229] T f1a904cb2fed48e3a1e60f1dca6d7537 P d68a40cd55a24edbb306ac354251f160 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:17:44.339329 23577 raft_consensus.cc:2258] T f1a904cb2fed48e3a1e60f1dca6d7537 P d68a40cd55a24edbb306ac354251f160 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:17:44.339439 23576 tablet_replica.cc:330] T f1a904cb2fed48e3a1e60f1dca6d7537 P f87970e42bbb4b10884b1606288565ca: stopping tablet replica
I20260929 03:17:44.339502 23576 raft_consensus.cc:2229] T f1a904cb2fed48e3a1e60f1dca6d7537 P f87970e42bbb4b10884b1606288565ca [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:17:44.339570 23576 raft_consensus.cc:2258] T f1a904cb2fed48e3a1e60f1dca6d7537 P f87970e42bbb4b10884b1606288565ca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:17:44.339598 23577 ts_tablet_manager.cc:1905] T f1a904cb2fed48e3a1e60f1dca6d7537 P d68a40cd55a24edbb306ac354251f160: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:17:44.339797 23576 ts_tablet_manager.cc:1905] T f1a904cb2fed48e3a1e60f1dca6d7537 P f87970e42bbb4b10884b1606288565ca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:17:44.341357 23577 ts_tablet_manager.cc:1918] T f1a904cb2fed48e3a1e60f1dca6d7537 P d68a40cd55a24edbb306ac354251f160: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 03:17:44.341444 23577 log.cc:1196] T f1a904cb2fed48e3a1e60f1dca6d7537 P d68a40cd55a24edbb306ac354251f160: Deleting WAL directory at /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651852476208-21193-0/minicluster-data/ts-1/wal/wals/f1a904cb2fed48e3a1e60f1dca6d7537
I20260929 03:17:44.341711 23577 ts_tablet_manager.cc:1939] T f1a904cb2fed48e3a1e60f1dca6d7537 P d68a40cd55a24edbb306ac354251f160: Deleting consensus metadata
I20260929 03:17:44.341722 23576 ts_tablet_manager.cc:1918] T f1a904cb2fed48e3a1e60f1dca6d7537 P f87970e42bbb4b10884b1606288565ca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 03:17:44.341774 23576 log.cc:1196] T f1a904cb2fed48e3a1e60f1dca6d7537 P f87970e42bbb4b10884b1606288565ca: Deleting WAL directory at /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651852476208-21193-0/minicluster-data/ts-2/wal/wals/f1a904cb2fed48e3a1e60f1dca6d7537
I20260929 03:17:44.341995 23576 ts_tablet_manager.cc:1939] T f1a904cb2fed48e3a1e60f1dca6d7537 P f87970e42bbb4b10884b1606288565ca: Deleting consensus metadata
I20260929 03:17:44.342072 22910 catalog_manager.cc:4841] TS d68a40cd55a24edbb306ac354251f160 (127.20.178.66:34631): tablet f1a904cb2fed48e3a1e60f1dca6d7537 (table test-table [id=b8723c594032412a97d2ccfc2ee4401f]) successfully deleted
I20260929 03:17:44.342339 22912 catalog_manager.cc:4841] TS f87970e42bbb4b10884b1606288565ca (127.20.178.67:35231): tablet f1a904cb2fed48e3a1e60f1dca6d7537 (table test-table [id=b8723c594032412a97d2ccfc2ee4401f]) successfully deleted
W20260929 03:17:44.344661 22913 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.20.178.65:43333 (error 108)
I20260929 03:17:44.344884 21193 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuoaHbX/build/release/bin/kudu with pid 23138
I20260929 03:17:44.349079 23579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:44.348837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.65:43333 (local address 127.0.0.1:33536)
0929 03:17:44.348971 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0929 03:17:44.349028 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.178.65:43333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":65,"threads_started":1}
W20260929 03:17:44.349237 22913 catalog_manager.cc:4584] TS 7502a5d6be1646cba535265e54b2f07e (127.20.178.65:43333): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f1a904cb2fed48e3a1e60f1dca6d7537: Network error: Client connection negotiation failed: client connection to 127.20.178.65:43333: connect: Connection refused (error 111)
I20260929 03:17:44.350908 21193 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuoaHbX/build/release/bin/kudu with pid 23278
I20260929 03:17:44.356933 21193 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuoaHbX/build/release/bin/kudu with pid 22894
2026-09-29T03:17:44Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (5717 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651852476208-21193-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:17:44 dist-test-slave-c9qh krb5kdc[23582](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:17:44 dist-test-slave-c9qh krb5kdc[23582](info): set up 2 sockets
Sep 29 03:17:44 dist-test-slave-c9qh krb5kdc[23582](info): commencing operation
krb5kdc: starting...
W20260929 03:17:45.380363 21193 mini_kdc.cc:121] Time spent starting KDC: real 1.002s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651852476208-21193-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651852476208-21193-0/krb5kdc/test-user.keytab.
Sep 29 03:17:45 dist-test-slave-c9qh krb5kdc[23582](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651865, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:17:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:17:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.178.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.178.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.178.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.126.keytab.
Entry for principal kudu/127.20.178.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.126.keytab.
I20260929 03:17:45.495723 21193 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuoaHbX/build/release/bin/kudu
/tmp/dist-test-taskuoaHbX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651852476208-21193-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651852476208-21193-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651852476208-21193-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651852476208-21193-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.178.126:45867
--webserver_interface=127.20.178.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.126.keytab
--principal=kudu/127.20.178.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.178.84:32959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.178.126:45867
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651852476208-21193-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651852476208-21193-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651852476208-21193-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:17:45.575109 23595 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:17:45.575304 23595 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:17:45.575325 23595 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:17:45.576471 23595 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:17:45.576515 23595 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:17:45.576529 23595 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:17:45.576572 23595 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:17:45.576602 23595 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:17:45.578102 23595 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.178.84:32959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651852476208-21193-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651852476208-21193-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.178.126:45867
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.126.keytab
--principal=kudu/127.20.178.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.178.126:45867
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651852476208-21193-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.178.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23595
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651852476208-21193-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:17:45.578330 23595 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:17:45.578568 23595 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:17:45.583670 23601 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:17:45.583714 23600 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:17:45.583803 23603 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:17:45.583899 23595 server_base.cc:1034] running on GCE node
I20260929 03:17:45.584142 23595 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:17:45.584442 23595 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:17:45.585610 23595 hybrid_clock.cc:648] HybridClock initialized: now 1790651865585607 us; error 48 us; skew 500 ppm
Sep 29 03:17:45 dist-test-slave-c9qh krb5kdc[23582](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651865, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:17:45.588193 23595 init.cc:377] Logged in from keytab as kudu/127.20.178.126@KRBTEST.COM (short username kudu)
I20260929 03:17:45.588711 23595 webserver.cc:460] Webserver started at http://127.20.178.126:35407/ using document root <none> and password file <none>
I20260929 03:17:45.588940 23595 fs_manager.cc:359] Metadata directory not provided
I20260929 03:17:45.589007 23595 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:17:45.589146 23595 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:17:45.590186 23595 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651852476208-21193-0/minicluster-data/master-0/data/instance:
uuid: "dad269b56d4a4c31bfd24eb68267b211"
format_stamp: "Formatted at 2026-09-29 03:17:45 on dist-test-slave-c9qh"
server_key: "1333b2e6a81aaef3d834c99a07bb512e"
server_key_iv: "c5172b1cef05e628180dcdf01b591250"
server_key_version: "encryptionkey@0"
I20260929 03:17:45.590554 23595 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651852476208-21193-0/minicluster-data/master-0/wal/instance:
uuid: "dad269b56d4a4c31bfd24eb68267b211"
format_stamp: "Formatted at 2026-09-29 03:17:45 on dist-test-slave-c9qh"
server_key: "1333b2e6a81aaef3d834c99a07bb512e"
server_key_iv: "c5172b1cef05e628180dcdf01b591250"
server_key_version: "encryptionkey@0"
I20260929 03:17:45.591913 23595 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 03:17:45.592720 23610 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:17:45.592900 23595 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260929 03:17:45.592998 23595 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651852476208-21193-0/minicluster-data/master-0/data,/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651852476208-21193-0/minicluster-data/master-0/wal
uuid: "dad269b56d4a4c31bfd24eb68267b211"
format_stamp: "Formatted at 2026-09-29 03:17:45 on dist-test-slave-c9qh"
server_key: "1333b2e6a81aaef3d834c99a07bb512e"
server_key_iv: "c5172b1cef05e628180dcdf01b591250"
server_key_version: "encryptionkey@0"
I20260929 03:17:45.593084 23595 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651852476208-21193-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651852476208-21193-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651852476208-21193-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:17:45.626631 23595 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:17:45.628554 23595 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:17:45.628764 23595 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:17:45.632807 23595 rpc_server.cc:307] RPC server started. Bound to: 127.20.178.126:45867
I20260929 03:17:45.632844 23662 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.178.126:45867 every 8 connection(s)
I20260929 03:17:45.633210 23595 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651852476208-21193-0/minicluster-data/master-0/data/info.pb
I20260929 03:17:45.634063 23663 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:17:45.636299 23663 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dad269b56d4a4c31bfd24eb68267b211: Bootstrap starting.
I20260929 03:17:45.636951 23663 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dad269b56d4a4c31bfd24eb68267b211: Neither blocks nor log segments found. Creating new log.
I20260929 03:17:45.637247 23663 log.cc:824] T 00000000000000000000000000000000 P dad269b56d4a4c31bfd24eb68267b211: Log is configured to *not* fsync() on all Append() calls
I20260929 03:17:45.637928 23663 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dad269b56d4a4c31bfd24eb68267b211: No bootstrap required, opened a new log
I20260929 03:17:45.639493 23663 raft_consensus.cc:348] T 00000000000000000000000000000000 P dad269b56d4a4c31bfd24eb68267b211 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dad269b56d4a4c31bfd24eb68267b211" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 45867 } }
I20260929 03:17:45.639631 23663 raft_consensus.cc:374] T 00000000000000000000000000000000 P dad269b56d4a4c31bfd24eb68267b211 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:17:45.639654 23663 raft_consensus.cc:729] T 00000000000000000000000000000000 P dad269b56d4a4c31bfd24eb68267b211 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dad269b56d4a4c31bfd24eb68267b211, State: Initialized, Role: FOLLOWER
I20260929 03:17:45.639813 23663 consensus_queue.cc:260] T 00000000000000000000000000000000 P dad269b56d4a4c31bfd24eb68267b211 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dad269b56d4a4c31bfd24eb68267b211" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 45867 } }
I20260929 03:17:45.639890 23663 raft_consensus.cc:388] T 00000000000000000000000000000000 P dad269b56d4a4c31bfd24eb68267b211 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:17:45.639912 23663 raft_consensus.cc:482] T 00000000000000000000000000000000 P dad269b56d4a4c31bfd24eb68267b211 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:17:45.639940 23663 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dad269b56d4a4c31bfd24eb68267b211 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:17:45.640476 23663 raft_consensus.cc:504] T 00000000000000000000000000000000 P dad269b56d4a4c31bfd24eb68267b211 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dad269b56d4a4c31bfd24eb68267b211" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 45867 } }
I20260929 03:17:45.640575 23663 leader_election.cc:302] T 00000000000000000000000000000000 P dad269b56d4a4c31bfd24eb68267b211 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dad269b56d4a4c31bfd24eb68267b211; no voters:
I20260929 03:17:45.640769 23663 leader_election.cc:288] T 00000000000000000000000000000000 P dad269b56d4a4c31bfd24eb68267b211 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:17:45.640813 21193 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuoaHbX/build/release/bin/kudu as pid 23595
I20260929 03:17:45.640827 23666 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dad269b56d4a4c31bfd24eb68267b211 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:17:45.640914 21193 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651852476208-21193-0/minicluster-data/master-0/wal/instance
I20260929 03:17:45.640974 23666 raft_consensus.cc:686] T 00000000000000000000000000000000 P dad269b56d4a4c31bfd24eb68267b211 [term 1 LEADER]: Becoming Leader. State: Replica: dad269b56d4a4c31bfd24eb68267b211, State: Running, Role: LEADER
I20260929 03:17:45.641028 23663 sys_catalog.cc:564] T 00000000000000000000000000000000 P dad269b56d4a4c31bfd24eb68267b211 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:17:45.641093 21193 external_mini_cluster.cc:1428] Setting key 391998cc823084d9f21ee3b02d917b04
I20260929 03:17:45.641108 23666 consensus_queue.cc:237] T 00000000000000000000000000000000 P dad269b56d4a4c31bfd24eb68267b211 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dad269b56d4a4c31bfd24eb68267b211" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 45867 } }
I20260929 03:17:45.641449 23667 sys_catalog.cc:455] T 00000000000000000000000000000000 P dad269b56d4a4c31bfd24eb68267b211 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dad269b56d4a4c31bfd24eb68267b211" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dad269b56d4a4c31bfd24eb68267b211" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 45867 } } }
I20260929 03:17:45.641536 23667 sys_catalog.cc:458] T 00000000000000000000000000000000 P dad269b56d4a4c31bfd24eb68267b211 [sys.catalog]: This master's current role is: LEADER
I20260929 03:17:45.641755 23671 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:17:45.641804 23668 sys_catalog.cc:455] T 00000000000000000000000000000000 P dad269b56d4a4c31bfd24eb68267b211 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dad269b56d4a4c31bfd24eb68267b211. Latest consensus state: current_term: 1 leader_uuid: "dad269b56d4a4c31bfd24eb68267b211" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dad269b56d4a4c31bfd24eb68267b211" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 45867 } } }
I20260929 03:17:45.641988 23668 sys_catalog.cc:458] T 00000000000000000000000000000000 P dad269b56d4a4c31bfd24eb68267b211 [sys.catalog]: This master's current role is: LEADER
I20260929 03:17:45.642256 23671 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:17:45.643972 23671 catalog_manager.cc:1347] Generated new cluster ID: bacdc0c0a3f146f9993d52fdb8d0449b
I20260929 03:17:45.644025 23671 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 29 03:17:45 dist-test-slave-c9qh krb5kdc[23582](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651865, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.178.126@KRBTEST.COM
I20260929 03:17:45.652020 23672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:45.641581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37184 (local address 127.20.178.126:45867)
0929 03:17:45.641741 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:45.641751 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:45.642362 (+ 611us) server_negotiation.cc:407] Connection header received
0929 03:17:45.642617 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:45.642628 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:45.642705 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:45.642832 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:17:45.643674 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:45.644736 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:45.645474 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:45.645889 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:45.646000 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:45.648325 (+ 2325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:17:45.648329 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:17:45.648329 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:17:45.648347 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:17:45.650495 (+ 2148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:45.650872 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:45.650875 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:45.650876 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:45.650983 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:45.651208 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:45.651209 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:45.651209 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:45.651441 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:17:45.651611 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0929 03:17:45.651716 (+ 105us) server_negotiation.cc:299] Negotiation successful
0929 03:17:45.651855 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":68,"threads_started":1}
I20260929 03:17:45.666348 23671 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:17:45.667115 23671 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:17:45.670392 23671 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dad269b56d4a4c31bfd24eb68267b211: Generated new TSK 0
I20260929 03:17:45.670555 23671 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.178.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.178.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.178.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.65.keytab.
Entry for principal kudu/127.20.178.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.65.keytab.
I20260929 03:17:45.699143 21193 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuoaHbX/build/release/bin/kudu
/tmp/dist-test-taskuoaHbX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651852476208-21193-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651852476208-21193-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651852476208-21193-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651852476208-21193-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.178.65:0
--local_ip_for_outbound_sockets=127.20.178.65
--webserver_interface=127.20.178.65
--webserver_port=0
--tserver_master_addrs=127.20.178.126:45867
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.65.keytab
--principal=kudu/127.20.178.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.178.84:32959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651852476208-21193-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651852476208-21193-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651852476208-21193-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:17:45.794299 23689 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:17:45.794500 23689 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:17:45.794525 23689 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:17:45.795841 23689 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:17:45.795882 23689 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:17:45.795910 23689 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.178.65
I20260929 03:17:45.797472 23689 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.178.84:32959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651852476208-21193-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651852476208-21193-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.65.keytab
--principal=kudu/127.20.178.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.178.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651852476208-21193-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.178.65
--webserver_port=0
--tserver_master_addrs=127.20.178.126:45867
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23689
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.178.65
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651852476208-21193-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:17:45.797724 23689 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:17:45.797967 23689 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:17:45.802912 23697 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:17:45.802923 23694 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:17:45.802920 23695 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:17:45.803181 23689 server_base.cc:1034] running on GCE node
I20260929 03:17:45.803370 23689 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:17:45.803589 23689 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:17:45.804692 23689 hybrid_clock.cc:648] HybridClock initialized: now 1790651865804675 us; error 32 us; skew 500 ppm
Sep 29 03:17:45 dist-test-slave-c9qh krb5kdc[23582](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651865, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:17:45.807096 23689 init.cc:377] Logged in from keytab as kudu/127.20.178.65@KRBTEST.COM (short username kudu)
I20260929 03:17:45.807580 23689 webserver.cc:460] Webserver started at http://127.20.178.65:35425/ using document root <none> and password file <none>
I20260929 03:17:45.807833 23689 fs_manager.cc:359] Metadata directory not provided
I20260929 03:17:45.807878 23689 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:17:45.808032 23689 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:17:45.809224 23689 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651852476208-21193-0/minicluster-data/ts-0/data/instance:
uuid: "95398bb91fe04cbaa75c430d299adb6a"
format_stamp: "Formatted at 2026-09-29 03:17:45 on dist-test-slave-c9qh"
server_key: "945a1df61b625a93d6210fc258cc5132"
server_key_iv: "75a6f53a5dfe2a4800070bdcb8051e0f"
server_key_version: "encryptionkey@0"
I20260929 03:17:45.809599 23689 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651852476208-21193-0/minicluster-data/ts-0/wal/instance:
uuid: "95398bb91fe04cbaa75c430d299adb6a"
format_stamp: "Formatted at 2026-09-29 03:17:45 on dist-test-slave-c9qh"
server_key: "945a1df61b625a93d6210fc258cc5132"
server_key_iv: "75a6f53a5dfe2a4800070bdcb8051e0f"
server_key_version: "encryptionkey@0"
I20260929 03:17:45.811267 23689 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 03:17:45.812291 23704 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:17:45.812456 23689 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 03:17:45.812548 23689 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651852476208-21193-0/minicluster-data/ts-0/data,/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651852476208-21193-0/minicluster-data/ts-0/wal
uuid: "95398bb91fe04cbaa75c430d299adb6a"
format_stamp: "Formatted at 2026-09-29 03:17:45 on dist-test-slave-c9qh"
server_key: "945a1df61b625a93d6210fc258cc5132"
server_key_iv: "75a6f53a5dfe2a4800070bdcb8051e0f"
server_key_version: "encryptionkey@0"
I20260929 03:17:45.812665 23689 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651852476208-21193-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651852476208-21193-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651852476208-21193-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:17:45.829929 23689 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:17:45.832237 23689 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:17:45.832348 23689 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:17:45.832542 23689 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:17:45.832888 23689 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:17:45.832923 23689 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:17:45.832945 23689 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:17:45.832958 23689 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:17:45.838524 23689 rpc_server.cc:307] RPC server started. Bound to: 127.20.178.65:33739
I20260929 03:17:45.838605 23817 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.178.65:33739 every 8 connection(s)
I20260929 03:17:45.838915 23689 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651852476208-21193-0/minicluster-data/ts-0/data/info.pb
I20260929 03:17:45.844672 21193 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuoaHbX/build/release/bin/kudu as pid 23689
I20260929 03:17:45.844785 21193 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651852476208-21193-0/minicluster-data/ts-0/wal/instance
I20260929 03:17:45.844981 21193 external_mini_cluster.cc:1428] Setting key be7037dc314870b9fc0b25e872e67b18
Sep 29 03:17:45 dist-test-slave-c9qh krb5kdc[23582](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651865, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.65@KRBTEST.COM for kudu/127.20.178.126@KRBTEST.COM
I20260929 03:17:45.848608 23672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:45.839648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.65:49457 (local address 127.20.178.126:45867)
0929 03:17:45.839713 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:45.839716 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:45.839943 (+ 227us) server_negotiation.cc:407] Connection header received
0929 03:17:45.840435 (+ 492us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:45.840436 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:45.840449 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:45.840514 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:17:45.841605 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:45.842159 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:45.843188 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:45.843379 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:45.843434 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:45.845782 (+ 2348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:17:45.845783 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:17:45.845784 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:17:45.845790 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:17:45.847253 (+ 1463us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:45.847633 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:45.847635 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:45.847635 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:45.847704 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:45.847944 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:45.847946 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:45.847946 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:45.848091 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:17:45.848157 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0929 03:17:45.848425 (+ 268us) server_negotiation.cc:299] Negotiation successful
0929 03:17:45.848518 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260929 03:17:45.848878 23820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:45.839629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:45867 (local address 127.20.178.65:49457)
0929 03:17:45.839809 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0929 03:17:45.839852 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 03:17:45.840325 (+ 473us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:17:45.840570 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:17:45.840574 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:17:45.841250 (+ 676us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:17:45.841509 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:45.841515 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:45.842259 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:45.842260 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:45.843110 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:45.843112 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:45.843456 (+ 344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:45.843457 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:45.843556 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:45.843998 (+ 442us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:17:45.844006 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:17:45.845663 (+ 1657us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:17:45.847338 (+ 1675us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:17:45.847344 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:17:45.847346 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:17:45.847556 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:17:45.847775 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:17:45.847775 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:17:45.847775 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:17:45.847872 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:17:45.848161 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:17:45.848165 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:17:45.848283 (+ 118us) client_negotiation.cc:769] Sending connection context
0929 03:17:45.848397 (+ 114us) client_negotiation.cc:240] Negotiation successful
0929 03:17:45.848548 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":95,"threads_started":1}
I20260929 03:17:45.849392 23818 heartbeater.cc:347] Connected to a master server at 127.20.178.126:45867
I20260929 03:17:45.849512 23818 heartbeater.cc:464] Registering TS with master...
I20260929 03:17:45.849692 23818 heartbeater.cc:511] Master 127.20.178.126:45867 requested a full tablet report, sending...
I20260929 03:17:45.850250 23627 ts_manager.cc:194] Registered new tserver with Master: 95398bb91fe04cbaa75c430d299adb6a (127.20.178.65:33739)
I20260929 03:17:45.850859 23627 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.178.65@KRBTEST.COM'} at 127.20.178.65:49457
WARNING: no policy specified for kudu/127.20.178.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.178.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.178.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.66.keytab.
Entry for principal kudu/127.20.178.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.66.keytab.
I20260929 03:17:45.871066 21193 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuoaHbX/build/release/bin/kudu
/tmp/dist-test-taskuoaHbX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651852476208-21193-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651852476208-21193-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651852476208-21193-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651852476208-21193-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.178.66:0
--local_ip_for_outbound_sockets=127.20.178.66
--webserver_interface=127.20.178.66
--webserver_port=0
--tserver_master_addrs=127.20.178.126:45867
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.66.keytab
--principal=kudu/127.20.178.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.178.84:32959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651852476208-21193-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651852476208-21193-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651852476208-21193-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:17:45.952423 23823 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:17:45.952736 23823 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:17:45.952766 23823 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:17:45.953975 23823 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:17:45.954020 23823 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:17:45.954046 23823 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.178.66
I20260929 03:17:45.955546 23823 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.178.84:32959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651852476208-21193-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651852476208-21193-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.66.keytab
--principal=kudu/127.20.178.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.178.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651852476208-21193-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.178.66
--webserver_port=0
--tserver_master_addrs=127.20.178.126:45867
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23823
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.178.66
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651852476208-21193-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:17:45.955794 23823 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:17:45.956041 23823 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:17:45.961627 23828 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:17:45.961597 23831 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:17:45.961603 23829 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:17:45.961889 23823 server_base.cc:1034] running on GCE node
I20260929 03:17:45.962129 23823 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:17:45.962394 23823 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:17:45.963583 23823 hybrid_clock.cc:648] HybridClock initialized: now 1790651865963577 us; error 38 us; skew 500 ppm
Sep 29 03:17:45 dist-test-slave-c9qh krb5kdc[23582](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651865, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:17:45.966167 23823 init.cc:377] Logged in from keytab as kudu/127.20.178.66@KRBTEST.COM (short username kudu)
I20260929 03:17:45.966622 23823 webserver.cc:460] Webserver started at http://127.20.178.66:43599/ using document root <none> and password file <none>
I20260929 03:17:45.966858 23823 fs_manager.cc:359] Metadata directory not provided
I20260929 03:17:45.966928 23823 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:17:45.967079 23823 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:17:45.968124 23823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651852476208-21193-0/minicluster-data/ts-1/data/instance:
uuid: "83db7fff98124048a9686e4dcb766bf3"
format_stamp: "Formatted at 2026-09-29 03:17:45 on dist-test-slave-c9qh"
server_key: "7d7e0acbb707e6b4b2aa5137be04828e"
server_key_iv: "2e60c41cf84003270786a8a354a28b67"
server_key_version: "encryptionkey@0"
I20260929 03:17:45.968510 23823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651852476208-21193-0/minicluster-data/ts-1/wal/instance:
uuid: "83db7fff98124048a9686e4dcb766bf3"
format_stamp: "Formatted at 2026-09-29 03:17:45 on dist-test-slave-c9qh"
server_key: "7d7e0acbb707e6b4b2aa5137be04828e"
server_key_iv: "2e60c41cf84003270786a8a354a28b67"
server_key_version: "encryptionkey@0"
I20260929 03:17:45.969870 23823 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260929 03:17:45.970606 23838 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:17:45.970775 23823 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260929 03:17:45.970881 23823 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651852476208-21193-0/minicluster-data/ts-1/data,/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651852476208-21193-0/minicluster-data/ts-1/wal
uuid: "83db7fff98124048a9686e4dcb766bf3"
format_stamp: "Formatted at 2026-09-29 03:17:45 on dist-test-slave-c9qh"
server_key: "7d7e0acbb707e6b4b2aa5137be04828e"
server_key_iv: "2e60c41cf84003270786a8a354a28b67"
server_key_version: "encryptionkey@0"
I20260929 03:17:45.970956 23823 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651852476208-21193-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651852476208-21193-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651852476208-21193-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:17:45.989413 23823 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:17:45.991268 23823 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:17:45.991441 23823 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:17:45.991664 23823 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:17:45.991976 23823 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:17:45.992031 23823 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:17:45.992074 23823 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:17:45.992102 23823 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:17:45.998364 23823 rpc_server.cc:307] RPC server started. Bound to: 127.20.178.66:44533
I20260929 03:17:45.998406 23951 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.178.66:44533 every 8 connection(s)
I20260929 03:17:45.998709 23823 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651852476208-21193-0/minicluster-data/ts-1/data/info.pb
Sep 29 03:17:46 dist-test-slave-c9qh krb5kdc[23582](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651865, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.66@KRBTEST.COM for kudu/127.20.178.126@KRBTEST.COM
I20260929 03:17:46.005863 21193 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuoaHbX/build/release/bin/kudu as pid 23823
I20260929 03:17:46.005978 21193 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651852476208-21193-0/minicluster-data/ts-1/wal/instance
I20260929 03:17:46.006206 21193 external_mini_cluster.cc:1428] Setting key 575420e19d2dcc9e98807b1d942ea8a4
I20260929 03:17:46.008105 23672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:45.999377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.66:37413 (local address 127.20.178.126:45867)
0929 03:17:45.999454 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:45.999456 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:45.999624 (+ 168us) server_negotiation.cc:407] Connection header received
0929 03:17:46.000096 (+ 472us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:46.000097 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:46.000106 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:46.000162 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:17:46.001123 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:46.001612 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:46.002553 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:46.002732 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:46.002786 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:46.004992 (+ 2206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:17:46.004993 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:17:46.004994 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:17:46.005001 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:17:46.006701 (+ 1700us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:46.007080 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:46.007082 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:46.007082 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:46.007148 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:46.007394 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:46.007395 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:46.007396 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:46.007550 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:17:46.007622 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0929 03:17:46.007871 (+ 249us) server_negotiation.cc:299] Negotiation successful
0929 03:17:46.007961 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 03:17:46.008180 23954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:45.999341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:45867 (local address 127.20.178.66:37413)
0929 03:17:45.999514 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0929 03:17:45.999554 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 03:17:46.000014 (+ 460us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:17:46.000221 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:17:46.000224 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:17:46.000853 (+ 629us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:17:46.001043 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:46.001048 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:46.001691 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:46.001691 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:46.002476 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:46.002478 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:46.002805 (+ 327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:46.002805 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:46.002907 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:46.003454 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:17:46.003462 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:17:46.004903 (+ 1441us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:17:46.006777 (+ 1874us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:17:46.006784 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:17:46.006787 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:17:46.007008 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:17:46.007217 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:17:46.007218 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:17:46.007218 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:17:46.007329 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:17:46.007615 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:17:46.007620 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:17:46.007733 (+ 113us) client_negotiation.cc:769] Sending connection context
0929 03:17:46.007842 (+ 109us) client_negotiation.cc:240] Negotiation successful
0929 03:17:46.007977 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":99,"threads_started":1}
I20260929 03:17:46.008700 23952 heartbeater.cc:347] Connected to a master server at 127.20.178.126:45867
I20260929 03:17:46.008809 23952 heartbeater.cc:464] Registering TS with master...
I20260929 03:17:46.009008 23952 heartbeater.cc:511] Master 127.20.178.126:45867 requested a full tablet report, sending...
I20260929 03:17:46.009385 23627 ts_manager.cc:194] Registered new tserver with Master: 83db7fff98124048a9686e4dcb766bf3 (127.20.178.66:44533)
I20260929 03:17:46.009754 23627 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.178.66@KRBTEST.COM'} at 127.20.178.66:37413
WARNING: no policy specified for kudu/127.20.178.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.178.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.178.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.67.keytab.
Entry for principal kudu/127.20.178.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.67.keytab.
I20260929 03:17:46.031327 21193 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuoaHbX/build/release/bin/kudu
/tmp/dist-test-taskuoaHbX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651852476208-21193-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651852476208-21193-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651852476208-21193-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651852476208-21193-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.178.67:0
--local_ip_for_outbound_sockets=127.20.178.67
--webserver_interface=127.20.178.67
--webserver_port=0
--tserver_master_addrs=127.20.178.126:45867
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.67.keytab
--principal=kudu/127.20.178.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.178.84:32959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651852476208-21193-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651852476208-21193-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651852476208-21193-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:17:46.111508 23957 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:17:46.111774 23957 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:17:46.111815 23957 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:17:46.113010 23957 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:17:46.113049 23957 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:17:46.113075 23957 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.178.67
I20260929 03:17:46.114406 23957 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.178.84:32959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651852476208-21193-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651852476208-21193-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.67.keytab
--principal=kudu/127.20.178.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.178.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651852476208-21193-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.178.67
--webserver_port=0
--tserver_master_addrs=127.20.178.126:45867
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23957
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.178.67
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651852476208-21193-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:17:46.114585 23957 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:17:46.114755 23957 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:17:46.119863 23963 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:17:46.119890 23965 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:17:46.119863 23962 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:17:46.119894 23957 server_base.cc:1034] running on GCE node
I20260929 03:17:46.120282 23957 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:17:46.120544 23957 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:17:46.121726 23957 hybrid_clock.cc:648] HybridClock initialized: now 1790651866121717 us; error 36 us; skew 500 ppm
Sep 29 03:17:46 dist-test-slave-c9qh krb5kdc[23582](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651866, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:17:46.124105 23957 init.cc:377] Logged in from keytab as kudu/127.20.178.67@KRBTEST.COM (short username kudu)
I20260929 03:17:46.124547 23957 webserver.cc:460] Webserver started at http://127.20.178.67:39557/ using document root <none> and password file <none>
I20260929 03:17:46.124811 23957 fs_manager.cc:359] Metadata directory not provided
I20260929 03:17:46.124883 23957 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:17:46.125031 23957 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:17:46.126068 23957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651852476208-21193-0/minicluster-data/ts-2/data/instance:
uuid: "bf2580981b704db1ab2fee1223eb153b"
format_stamp: "Formatted at 2026-09-29 03:17:46 on dist-test-slave-c9qh"
server_key: "24fc3b477837143a039213c8a6bc3b13"
server_key_iv: "de3076042fdf0219097d50ff535b77aa"
server_key_version: "encryptionkey@0"
I20260929 03:17:46.126426 23957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651852476208-21193-0/minicluster-data/ts-2/wal/instance:
uuid: "bf2580981b704db1ab2fee1223eb153b"
format_stamp: "Formatted at 2026-09-29 03:17:46 on dist-test-slave-c9qh"
server_key: "24fc3b477837143a039213c8a6bc3b13"
server_key_iv: "de3076042fdf0219097d50ff535b77aa"
server_key_version: "encryptionkey@0"
I20260929 03:17:46.127693 23957 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 03:17:46.128400 23972 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:17:46.128566 23957 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 03:17:46.128681 23957 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651852476208-21193-0/minicluster-data/ts-2/data,/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651852476208-21193-0/minicluster-data/ts-2/wal
uuid: "bf2580981b704db1ab2fee1223eb153b"
format_stamp: "Formatted at 2026-09-29 03:17:46 on dist-test-slave-c9qh"
server_key: "24fc3b477837143a039213c8a6bc3b13"
server_key_iv: "de3076042fdf0219097d50ff535b77aa"
server_key_version: "encryptionkey@0"
I20260929 03:17:46.128778 23957 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651852476208-21193-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651852476208-21193-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651852476208-21193-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:17:46.137748 23957 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:17:46.139400 23957 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:17:46.139501 23957 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:17:46.139659 23957 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:17:46.139959 23957 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:17:46.139993 23957 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:17:46.140038 23957 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:17:46.140064 23957 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:17:46.145982 23957 rpc_server.cc:307] RPC server started. Bound to: 127.20.178.67:32837
I20260929 03:17:46.146047 24085 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.178.67:32837 every 8 connection(s)
I20260929 03:17:46.146373 23957 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651852476208-21193-0/minicluster-data/ts-2/data/info.pb
Sep 29 03:17:46 dist-test-slave-c9qh krb5kdc[23582](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651866, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.67@KRBTEST.COM for kudu/127.20.178.126@KRBTEST.COM
I20260929 03:17:46.154804 23672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:46.147041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.67:55715 (local address 127.20.178.126:45867)
0929 03:17:46.147112 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:46.147114 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:46.147308 (+ 194us) server_negotiation.cc:407] Connection header received
0929 03:17:46.147730 (+ 422us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:46.147731 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:46.147741 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:46.147791 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:17:46.148708 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:46.149174 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:46.150104 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:46.150284 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:46.150333 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:46.152201 (+ 1868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:17:46.152203 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:17:46.152203 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:17:46.152211 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:17:46.153444 (+ 1233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:46.153846 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:46.153848 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:46.153848 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:46.153910 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:46.154155 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:46.154156 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:46.154156 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:46.154274 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:17:46.154353 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0929 03:17:46.154622 (+ 269us) server_negotiation.cc:299] Negotiation successful
0929 03:17:46.154704 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260929 03:17:46.154949 24088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:46.147021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:45867 (local address 127.20.178.67:55715)
0929 03:17:46.147189 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0929 03:17:46.147222 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0929 03:17:46.147654 (+ 432us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:17:46.147842 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:17:46.147845 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:17:46.148406 (+ 561us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:17:46.148583 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:46.148587 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:46.149269 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:46.149270 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:46.150033 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:46.150035 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:46.150354 (+ 319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:46.150355 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:46.150446 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:46.150843 (+ 397us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:17:46.150851 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:17:46.152094 (+ 1243us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:17:46.153529 (+ 1435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:17:46.153535 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:17:46.153537 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:17:46.153751 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:17:46.153983 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:17:46.153984 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:17:46.153984 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:17:46.154088 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:17:46.154345 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:17:46.154351 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:17:46.154486 (+ 135us) client_negotiation.cc:769] Sending connection context
0929 03:17:46.154587 (+ 101us) client_negotiation.cc:240] Negotiation successful
0929 03:17:46.154731 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":95,"threads_started":1}
I20260929 03:17:46.155400 24086 heartbeater.cc:347] Connected to a master server at 127.20.178.126:45867
I20260929 03:17:46.155474 24086 heartbeater.cc:464] Registering TS with master...
I20260929 03:17:46.155599 24086 heartbeater.cc:511] Master 127.20.178.126:45867 requested a full tablet report, sending...
I20260929 03:17:46.155916 23627 ts_manager.cc:194] Registered new tserver with Master: bf2580981b704db1ab2fee1223eb153b (127.20.178.67:32837)
I20260929 03:17:46.155987 21193 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuoaHbX/build/release/bin/kudu as pid 23957
I20260929 03:17:46.156067 21193 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651852476208-21193-0/minicluster-data/ts-2/wal/instance
I20260929 03:17:46.156234 21193 external_mini_cluster.cc:1428] Setting key 0ed6116d521d3e1029b839e28c961139
I20260929 03:17:46.156369 23627 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.178.67@KRBTEST.COM'} at 127.20.178.67:55715
I20260929 03:17:46.157047 21193 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260929 03:17:46.167563 23672 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:17:46.166651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37198 (local address 127.20.178.126:45867)
0929 03:17:46.166721 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:46.166723 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:46.166833 (+ 110us) server_negotiation.cc:407] Connection header received
0929 03:17:46.166917 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:46.166919 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:46.166929 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:46.166986 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:17:46.167491 (+ 505us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37198: BlockingRecv error: recv got EOF from 127.0.0.1:37198 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
I20260929 03:17:46.168596 21193 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuoaHbX/build/release/bin/kudu with pid 23689
I20260929 03:17:46.173324 21193 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuoaHbX/build/release/bin/kudu with pid 23823
I20260929 03:17:46.177779 21193 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuoaHbX/build/release/bin/kudu with pid 23957
I20260929 03:17:46.181996 21193 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuoaHbX/build/release/bin/kudu with pid 23595
2026-09-29T03:17:46Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (1823 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790651852476208-21193-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:17:46 dist-test-slave-c9qh krb5kdc[24098](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:17:46 dist-test-slave-c9qh krb5kdc[24098](info): set up 2 sockets
Sep 29 03:17:46 dist-test-slave-c9qh krb5kdc[24098](info): commencing operation
krb5kdc: starting...
W20260929 03:17:47.220366 21193 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790651852476208-21193-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790651852476208-21193-0/krb5kdc/test-user.keytab.
Sep 29 03:17:47 dist-test-slave-c9qh krb5kdc[24098](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651867, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:17:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:17:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.178.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.178.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.178.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.126.keytab.
Entry for principal kudu/127.20.178.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.126.keytab.
I20260929 03:17:47.335561 21193 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuoaHbX/build/release/bin/kudu
/tmp/dist-test-taskuoaHbX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790651852476208-21193-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790651852476208-21193-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790651852476208-21193-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790651852476208-21193-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.178.126:42849
--webserver_interface=127.20.178.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.126.keytab
--principal=kudu/127.20.178.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.178.84:40987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.178.126:42849
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790651852476208-21193-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790651852476208-21193-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790651852476208-21193-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:17:47.415448 24111 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:17:47.415658 24111 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:17:47.415688 24111 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:17:47.416888 24111 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:17:47.416931 24111 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:17:47.416946 24111 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:17:47.416958 24111 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:17:47.417003 24111 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:17:47.418527 24111 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.178.84:40987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790651852476208-21193-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790651852476208-21193-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.178.126:42849
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.126.keytab
--principal=kudu/127.20.178.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.178.126:42849
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790651852476208-21193-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.178.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24111
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790651852476208-21193-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:17:47.418804 24111 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:17:47.419036 24111 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:17:47.423831 24116 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:17:47.423828 24117 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:17:47.423985 24111 server_base.cc:1034] running on GCE node
W20260929 03:17:47.423828 24119 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:17:47.424301 24111 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:17:47.424557 24111 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:17:47.425701 24111 hybrid_clock.cc:648] HybridClock initialized: now 1790651867425685 us; error 28 us; skew 500 ppm
Sep 29 03:17:47 dist-test-slave-c9qh krb5kdc[24098](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651867, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:17:47.428138 24111 init.cc:377] Logged in from keytab as kudu/127.20.178.126@KRBTEST.COM (short username kudu)
I20260929 03:17:47.428541 24111 webserver.cc:460] Webserver started at http://127.20.178.126:44983/ using document root <none> and password file <none>
I20260929 03:17:47.428807 24111 fs_manager.cc:359] Metadata directory not provided
I20260929 03:17:47.428855 24111 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:17:47.428948 24111 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:17:47.430013 24111 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790651852476208-21193-0/minicluster-data/master-0/data/instance:
uuid: "d7c16c7914fd4bcc88e5d2dc590140c5"
format_stamp: "Formatted at 2026-09-29 03:17:47 on dist-test-slave-c9qh"
server_key: "4bb7cfe7c6f4a0eb8fc56ca7123d41c8"
server_key_iv: "b7289e394fa3ecf646ee0d8c4caed4a1"
server_key_version: "encryptionkey@0"
I20260929 03:17:47.430363 24111 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790651852476208-21193-0/minicluster-data/master-0/wal/instance:
uuid: "d7c16c7914fd4bcc88e5d2dc590140c5"
format_stamp: "Formatted at 2026-09-29 03:17:47 on dist-test-slave-c9qh"
server_key: "4bb7cfe7c6f4a0eb8fc56ca7123d41c8"
server_key_iv: "b7289e394fa3ecf646ee0d8c4caed4a1"
server_key_version: "encryptionkey@0"
I20260929 03:17:47.431653 24111 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 03:17:47.432367 24126 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:17:47.432546 24111 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 03:17:47.432664 24111 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790651852476208-21193-0/minicluster-data/master-0/data,/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790651852476208-21193-0/minicluster-data/master-0/wal
uuid: "d7c16c7914fd4bcc88e5d2dc590140c5"
format_stamp: "Formatted at 2026-09-29 03:17:47 on dist-test-slave-c9qh"
server_key: "4bb7cfe7c6f4a0eb8fc56ca7123d41c8"
server_key_iv: "b7289e394fa3ecf646ee0d8c4caed4a1"
server_key_version: "encryptionkey@0"
I20260929 03:17:47.432744 24111 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790651852476208-21193-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790651852476208-21193-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790651852476208-21193-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:17:47.447583 24111 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:17:47.449766 24111 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:17:47.449879 24111 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:17:47.453619 24111 rpc_server.cc:307] RPC server started. Bound to: 127.20.178.126:42849
I20260929 03:17:47.453704 24178 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.178.126:42849 every 8 connection(s)
I20260929 03:17:47.454181 24111 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790651852476208-21193-0/minicluster-data/master-0/data/info.pb
I20260929 03:17:47.454926 24179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:17:47.457322 24179 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7c16c7914fd4bcc88e5d2dc590140c5: Bootstrap starting.
I20260929 03:17:47.457942 24179 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d7c16c7914fd4bcc88e5d2dc590140c5: Neither blocks nor log segments found. Creating new log.
I20260929 03:17:47.458240 24179 log.cc:824] T 00000000000000000000000000000000 P d7c16c7914fd4bcc88e5d2dc590140c5: Log is configured to *not* fsync() on all Append() calls
I20260929 03:17:47.458941 24179 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7c16c7914fd4bcc88e5d2dc590140c5: No bootstrap required, opened a new log
I20260929 03:17:47.460614 21193 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuoaHbX/build/release/bin/kudu as pid 24111
I20260929 03:17:47.460726 21193 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790651852476208-21193-0/minicluster-data/master-0/wal/instance
I20260929 03:17:47.460634 24179 raft_consensus.cc:348] T 00000000000000000000000000000000 P d7c16c7914fd4bcc88e5d2dc590140c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7c16c7914fd4bcc88e5d2dc590140c5" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 42849 } }
I20260929 03:17:47.460863 24179 raft_consensus.cc:374] T 00000000000000000000000000000000 P d7c16c7914fd4bcc88e5d2dc590140c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:17:47.460903 24179 raft_consensus.cc:729] T 00000000000000000000000000000000 P d7c16c7914fd4bcc88e5d2dc590140c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7c16c7914fd4bcc88e5d2dc590140c5, State: Initialized, Role: FOLLOWER
I20260929 03:17:47.460933 21193 external_mini_cluster.cc:1428] Setting key 619de5cdecde8ac1a5ef468d38176be2
I20260929 03:17:47.461064 24179 consensus_queue.cc:260] T 00000000000000000000000000000000 P d7c16c7914fd4bcc88e5d2dc590140c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7c16c7914fd4bcc88e5d2dc590140c5" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 42849 } }
I20260929 03:17:47.461153 24179 raft_consensus.cc:388] T 00000000000000000000000000000000 P d7c16c7914fd4bcc88e5d2dc590140c5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:17:47.461207 24179 raft_consensus.cc:482] T 00000000000000000000000000000000 P d7c16c7914fd4bcc88e5d2dc590140c5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:17:47.461263 24179 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d7c16c7914fd4bcc88e5d2dc590140c5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:17:47.462036 24179 raft_consensus.cc:504] T 00000000000000000000000000000000 P d7c16c7914fd4bcc88e5d2dc590140c5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7c16c7914fd4bcc88e5d2dc590140c5" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 42849 } }
I20260929 03:17:47.462137 24179 leader_election.cc:302] T 00000000000000000000000000000000 P d7c16c7914fd4bcc88e5d2dc590140c5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d7c16c7914fd4bcc88e5d2dc590140c5; no voters:
I20260929 03:17:47.462352 24179 leader_election.cc:288] T 00000000000000000000000000000000 P d7c16c7914fd4bcc88e5d2dc590140c5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:17:47.462379 24184 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d7c16c7914fd4bcc88e5d2dc590140c5 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:17:47.462489 24184 raft_consensus.cc:686] T 00000000000000000000000000000000 P d7c16c7914fd4bcc88e5d2dc590140c5 [term 1 LEADER]: Becoming Leader. State: Replica: d7c16c7914fd4bcc88e5d2dc590140c5, State: Running, Role: LEADER
I20260929 03:17:47.462569 24184 consensus_queue.cc:237] T 00000000000000000000000000000000 P d7c16c7914fd4bcc88e5d2dc590140c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7c16c7914fd4bcc88e5d2dc590140c5" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 42849 } }
I20260929 03:17:47.462625 24179 sys_catalog.cc:564] T 00000000000000000000000000000000 P d7c16c7914fd4bcc88e5d2dc590140c5 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:17:47.462911 24184 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7c16c7914fd4bcc88e5d2dc590140c5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d7c16c7914fd4bcc88e5d2dc590140c5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7c16c7914fd4bcc88e5d2dc590140c5" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 42849 } } }
I20260929 03:17:47.462985 24184 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7c16c7914fd4bcc88e5d2dc590140c5 [sys.catalog]: This master's current role is: LEADER
I20260929 03:17:47.463106 24184 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7c16c7914fd4bcc88e5d2dc590140c5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d7c16c7914fd4bcc88e5d2dc590140c5. Latest consensus state: current_term: 1 leader_uuid: "d7c16c7914fd4bcc88e5d2dc590140c5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7c16c7914fd4bcc88e5d2dc590140c5" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 42849 } } }
I20260929 03:17:47.463176 24184 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7c16c7914fd4bcc88e5d2dc590140c5 [sys.catalog]: This master's current role is: LEADER
I20260929 03:17:47.463308 24189 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:17:47.464107 24189 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:17:47.465926 24189 catalog_manager.cc:1347] Generated new cluster ID: 09869143025d41c9a64a375684d510b3
I20260929 03:17:47.465984 24189 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 29 03:17:47 dist-test-slave-c9qh krb5kdc[24098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651867, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.178.126@KRBTEST.COM
I20260929 03:17:47.473019 24183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:47.461407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45320 (local address 127.20.178.126:42849)
0929 03:17:47.462068 (+ 661us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:47.462076 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:47.462129 (+ 53us) server_negotiation.cc:407] Connection header received
0929 03:17:47.462289 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:47.462301 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:47.462346 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:47.462494 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:17:47.463446 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:47.464887 (+ 1441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:47.465953 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:47.466672 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:47.466815 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:47.469236 (+ 2421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:17:47.469239 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:17:47.469240 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:17:47.469259 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:17:47.471436 (+ 2177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:47.471831 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:47.471835 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:47.471835 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:47.471929 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:47.472142 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:47.472143 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:47.472143 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:47.472401 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:17:47.472594 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0929 03:17:47.472690 (+ 96us) server_negotiation.cc:299] Negotiation successful
0929 03:17:47.472838 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":609,"thread_start_us":564,"threads_started":1}
I20260929 03:17:47.473942 21193 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 03:17:47.480947 24189 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:17:47.481576 24189 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:17:47.486294 24189 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d7c16c7914fd4bcc88e5d2dc590140c5: Generated new TSK 0
I20260929 03:17:47.486495 24189 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 03:17:47.494033 24183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:47.490023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45334 (local address 127.20.178.126:42849)
0929 03:17:47.490093 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:47.490095 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:47.490251 (+ 156us) server_negotiation.cc:407] Connection header received
0929 03:17:47.490317 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:47.490318 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:47.490330 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:47.490426 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:17:47.490784 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:47.491584 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:47.492331 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:47.492679 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:47.492748 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:47.493456 (+ 708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:17:47.493457 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:17:47.493458 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:17:47.493718 (+ 260us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:17:47.493739 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:17:47.493765 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:17:47.493835 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0929 03:17:47.493897 (+ 62us) server_negotiation.cc:299] Negotiation successful
0929 03:17:47.493943 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260929 03:17:47.496662 21193 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuoaHbX/build/release/bin/kudu with pid 24111
2026-09-29T03:17:47Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1312 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651852476208-21193-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:17:47 dist-test-slave-c9qh krb5kdc[24212](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:17:47 dist-test-slave-c9qh krb5kdc[24212](info): set up 2 sockets
Sep 29 03:17:47 dist-test-slave-c9qh krb5kdc[24212](info): commencing operation
krb5kdc: starting...
W20260929 03:17:48.532584 21193 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651852476208-21193-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651852476208-21193-0/krb5kdc/test-user.keytab.
Sep 29 03:17:48 dist-test-slave-c9qh krb5kdc[24212](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651868, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:17:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:17:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.178.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.178.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.178.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.126.keytab.
Entry for principal kudu/127.20.178.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.126.keytab.
I20260929 03:17:48.650983 21193 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuoaHbX/build/release/bin/kudu
/tmp/dist-test-taskuoaHbX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651852476208-21193-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651852476208-21193-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651852476208-21193-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651852476208-21193-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.178.126:40105
--webserver_interface=127.20.178.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.126.keytab
--principal=kudu/127.20.178.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.178.84:43339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.178.126:40105
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651852476208-21193-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651852476208-21193-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651852476208-21193-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:17:48.731118 24225 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:17:48.731369 24225 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:17:48.731406 24225 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:17:48.732615 24225 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:17:48.732733 24225 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:17:48.732760 24225 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:17:48.732791 24225 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:17:48.732818 24225 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:17:48.734303 24225 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.178.84:43339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651852476208-21193-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651852476208-21193-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.178.126:40105
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.126.keytab
--principal=kudu/127.20.178.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.178.126:40105
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651852476208-21193-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.178.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24225
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651852476208-21193-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:17:48.734576 24225 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:17:48.734822 24225 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:17:48.739687 24233 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:17:48.739697 24231 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:17:48.739692 24230 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:17:48.739871 24225 server_base.cc:1034] running on GCE node
I20260929 03:17:48.740080 24225 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:17:48.740375 24225 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:17:48.741537 24225 hybrid_clock.cc:648] HybridClock initialized: now 1790651868741524 us; error 44 us; skew 500 ppm
Sep 29 03:17:48 dist-test-slave-c9qh krb5kdc[24212](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651868, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:17:48.749879 24225 init.cc:377] Logged in from keytab as kudu/127.20.178.126@KRBTEST.COM (short username kudu)
I20260929 03:17:48.750298 24225 webserver.cc:460] Webserver started at http://127.20.178.126:34775/ using document root <none> and password file <none>
I20260929 03:17:48.750499 24225 fs_manager.cc:359] Metadata directory not provided
I20260929 03:17:48.750538 24225 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:17:48.750626 24225 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:17:48.751645 24225 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651852476208-21193-0/minicluster-data/master-0/data/instance:
uuid: "5c84d2edabb64fbab0e1fea2b185c0cd"
format_stamp: "Formatted at 2026-09-29 03:17:48 on dist-test-slave-c9qh"
server_key: "69fb70cfbd1d904e37adc1397ad046ec"
server_key_iv: "05898ae699598be727f6fac1c800ecc6"
server_key_version: "encryptionkey@0"
I20260929 03:17:48.751948 24225 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651852476208-21193-0/minicluster-data/master-0/wal/instance:
uuid: "5c84d2edabb64fbab0e1fea2b185c0cd"
format_stamp: "Formatted at 2026-09-29 03:17:48 on dist-test-slave-c9qh"
server_key: "69fb70cfbd1d904e37adc1397ad046ec"
server_key_iv: "05898ae699598be727f6fac1c800ecc6"
server_key_version: "encryptionkey@0"
I20260929 03:17:48.753328 24225 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260929 03:17:48.754004 24240 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:17:48.754194 24225 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 03:17:48.754279 24225 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651852476208-21193-0/minicluster-data/master-0/data,/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651852476208-21193-0/minicluster-data/master-0/wal
uuid: "5c84d2edabb64fbab0e1fea2b185c0cd"
format_stamp: "Formatted at 2026-09-29 03:17:48 on dist-test-slave-c9qh"
server_key: "69fb70cfbd1d904e37adc1397ad046ec"
server_key_iv: "05898ae699598be727f6fac1c800ecc6"
server_key_version: "encryptionkey@0"
I20260929 03:17:48.754334 24225 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651852476208-21193-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651852476208-21193-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651852476208-21193-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:17:48.765434 24225 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:17:48.767134 24225 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:17:48.767238 24225 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:17:48.771136 24225 rpc_server.cc:307] RPC server started. Bound to: 127.20.178.126:40105
I20260929 03:17:48.771188 24292 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.178.126:40105 every 8 connection(s)
I20260929 03:17:48.771528 24225 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651852476208-21193-0/minicluster-data/master-0/data/info.pb
I20260929 03:17:48.772498 24293 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:17:48.774788 24293 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c84d2edabb64fbab0e1fea2b185c0cd: Bootstrap starting.
I20260929 03:17:48.775353 24293 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5c84d2edabb64fbab0e1fea2b185c0cd: Neither blocks nor log segments found. Creating new log.
I20260929 03:17:48.775581 21193 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuoaHbX/build/release/bin/kudu as pid 24225
I20260929 03:17:48.775683 21193 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651852476208-21193-0/minicluster-data/master-0/wal/instance
I20260929 03:17:48.775699 24293 log.cc:824] T 00000000000000000000000000000000 P 5c84d2edabb64fbab0e1fea2b185c0cd: Log is configured to *not* fsync() on all Append() calls
I20260929 03:17:48.775857 21193 external_mini_cluster.cc:1428] Setting key 43d15ae59737ba641d87eb1350fa6cc6
I20260929 03:17:48.776499 24293 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c84d2edabb64fbab0e1fea2b185c0cd: No bootstrap required, opened a new log
I20260929 03:17:48.779050 24293 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5c84d2edabb64fbab0e1fea2b185c0cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c84d2edabb64fbab0e1fea2b185c0cd" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 40105 } }
I20260929 03:17:48.779217 24293 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5c84d2edabb64fbab0e1fea2b185c0cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:17:48.779255 24293 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5c84d2edabb64fbab0e1fea2b185c0cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c84d2edabb64fbab0e1fea2b185c0cd, State: Initialized, Role: FOLLOWER
I20260929 03:17:48.779373 24293 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5c84d2edabb64fbab0e1fea2b185c0cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c84d2edabb64fbab0e1fea2b185c0cd" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 40105 } }
I20260929 03:17:48.779457 24293 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5c84d2edabb64fbab0e1fea2b185c0cd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:17:48.779485 24293 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5c84d2edabb64fbab0e1fea2b185c0cd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:17:48.779527 24293 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5c84d2edabb64fbab0e1fea2b185c0cd [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:17:48.780153 24293 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5c84d2edabb64fbab0e1fea2b185c0cd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c84d2edabb64fbab0e1fea2b185c0cd" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 40105 } }
I20260929 03:17:48.780278 24293 leader_election.cc:302] T 00000000000000000000000000000000 P 5c84d2edabb64fbab0e1fea2b185c0cd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5c84d2edabb64fbab0e1fea2b185c0cd; no voters:
I20260929 03:17:48.780472 24293 leader_election.cc:288] T 00000000000000000000000000000000 P 5c84d2edabb64fbab0e1fea2b185c0cd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:17:48.780496 24298 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5c84d2edabb64fbab0e1fea2b185c0cd [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:17:48.780601 24298 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5c84d2edabb64fbab0e1fea2b185c0cd [term 1 LEADER]: Becoming Leader. State: Replica: 5c84d2edabb64fbab0e1fea2b185c0cd, State: Running, Role: LEADER
I20260929 03:17:48.780733 24298 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5c84d2edabb64fbab0e1fea2b185c0cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c84d2edabb64fbab0e1fea2b185c0cd" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 40105 } }
I20260929 03:17:48.780769 24293 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5c84d2edabb64fbab0e1fea2b185c0cd [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:17:48.781085 24300 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c84d2edabb64fbab0e1fea2b185c0cd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5c84d2edabb64fbab0e1fea2b185c0cd. Latest consensus state: current_term: 1 leader_uuid: "5c84d2edabb64fbab0e1fea2b185c0cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c84d2edabb64fbab0e1fea2b185c0cd" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 40105 } } }
I20260929 03:17:48.781175 24300 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c84d2edabb64fbab0e1fea2b185c0cd [sys.catalog]: This master's current role is: LEADER
I20260929 03:17:48.781086 24299 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c84d2edabb64fbab0e1fea2b185c0cd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5c84d2edabb64fbab0e1fea2b185c0cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c84d2edabb64fbab0e1fea2b185c0cd" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 40105 } } }
I20260929 03:17:48.781462 24299 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c84d2edabb64fbab0e1fea2b185c0cd [sys.catalog]: This master's current role is: LEADER
I20260929 03:17:48.781749 24302 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:17:48.782357 24302 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 29 03:17:48 dist-test-slave-c9qh krb5kdc[24212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651868, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.178.126@KRBTEST.COM
I20260929 03:17:48.783969 24302 catalog_manager.cc:1347] Generated new cluster ID: 307e87c657de495e92f4a295769389e5
I20260929 03:17:48.784016 24302 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:17:48.787412 24297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:48.776313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45548 (local address 127.20.178.126:40105)
0929 03:17:48.776510 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:48.776519 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:48.776543 (+ 24us) server_negotiation.cc:407] Connection header received
0929 03:17:48.777020 (+ 477us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:48.777027 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:48.777074 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:48.777187 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:17:48.777960 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:48.778809 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:48.779898 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:48.780406 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:48.780578 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:48.783736 (+ 3158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:17:48.783740 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:17:48.783740 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:17:48.783751 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:17:48.785762 (+ 2011us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:48.786215 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:48.786217 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:48.786218 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:48.786313 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:48.786542 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:48.786543 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:48.786543 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:48.786814 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:17:48.786956 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0929 03:17:48.787102 (+ 146us) server_negotiation.cc:299] Negotiation successful
0929 03:17:48.787220 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":114,"threads_started":1}
I20260929 03:17:48.802834 24302 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:17:48.803336 24302 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:17:48.805114 24302 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5c84d2edabb64fbab0e1fea2b185c0cd: Generated new TSK 0
I20260929 03:17:48.805286 24302 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.178.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.178.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.178.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.65.keytab.
Entry for principal kudu/127.20.178.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.65.keytab.
I20260929 03:17:48.835634 21193 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuoaHbX/build/release/bin/kudu
/tmp/dist-test-taskuoaHbX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651852476208-21193-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651852476208-21193-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651852476208-21193-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651852476208-21193-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.178.65:0
--local_ip_for_outbound_sockets=127.20.178.65
--webserver_interface=127.20.178.65
--webserver_port=0
--tserver_master_addrs=127.20.178.126:40105
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.65.keytab
--principal=kudu/127.20.178.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.178.84:43339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651852476208-21193-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651852476208-21193-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651852476208-21193-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:17:48.918056 24319 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:17:48.918290 24319 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:17:48.918332 24319 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:17:48.919934 24319 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:17:48.919993 24319 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:17:48.920042 24319 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.178.65
I20260929 03:17:48.921675 24319 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.178.84:43339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651852476208-21193-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651852476208-21193-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.65.keytab
--principal=kudu/127.20.178.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.178.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651852476208-21193-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.178.65
--webserver_port=0
--tserver_master_addrs=127.20.178.126:40105
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24319
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.178.65
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651852476208-21193-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:17:48.921931 24319 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:17:48.922155 24319 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:17:48.928109 24327 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:17:48.928150 24325 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:17:48.928170 24324 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:17:48.928737 24319 server_base.cc:1034] running on GCE node
I20260929 03:17:48.928911 24319 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:17:48.929229 24319 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:17:48.930413 24319 hybrid_clock.cc:648] HybridClock initialized: now 1790651868930407 us; error 49 us; skew 500 ppm
Sep 29 03:17:48 dist-test-slave-c9qh krb5kdc[24212](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651868, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:17:48.932888 24319 init.cc:377] Logged in from keytab as kudu/127.20.178.65@KRBTEST.COM (short username kudu)
I20260929 03:17:48.933329 24319 webserver.cc:460] Webserver started at http://127.20.178.65:33893/ using document root <none> and password file <none>
I20260929 03:17:48.933576 24319 fs_manager.cc:359] Metadata directory not provided
I20260929 03:17:48.933645 24319 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:17:48.933789 24319 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:17:48.934813 24319 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651852476208-21193-0/minicluster-data/ts-0/data/instance:
uuid: "2b48d954c83c4ffbb3b7e795d841089b"
format_stamp: "Formatted at 2026-09-29 03:17:48 on dist-test-slave-c9qh"
server_key: "5c89c075ff8dfae64a12f73b9303af2e"
server_key_iv: "c4b405c488eeea4e4160fb31c19232cc"
server_key_version: "encryptionkey@0"
I20260929 03:17:48.935173 24319 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651852476208-21193-0/minicluster-data/ts-0/wal/instance:
uuid: "2b48d954c83c4ffbb3b7e795d841089b"
format_stamp: "Formatted at 2026-09-29 03:17:48 on dist-test-slave-c9qh"
server_key: "5c89c075ff8dfae64a12f73b9303af2e"
server_key_iv: "c4b405c488eeea4e4160fb31c19232cc"
server_key_version: "encryptionkey@0"
I20260929 03:17:48.936463 24319 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 03:17:48.937260 24334 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:17:48.937451 24319 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260929 03:17:48.937548 24319 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651852476208-21193-0/minicluster-data/ts-0/data,/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651852476208-21193-0/minicluster-data/ts-0/wal
uuid: "2b48d954c83c4ffbb3b7e795d841089b"
format_stamp: "Formatted at 2026-09-29 03:17:48 on dist-test-slave-c9qh"
server_key: "5c89c075ff8dfae64a12f73b9303af2e"
server_key_iv: "c4b405c488eeea4e4160fb31c19232cc"
server_key_version: "encryptionkey@0"
I20260929 03:17:48.937631 24319 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651852476208-21193-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651852476208-21193-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651852476208-21193-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:17:48.964197 24319 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:17:48.966338 24319 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:17:48.966518 24319 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:17:48.966775 24319 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:17:48.967109 24319 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:17:48.967165 24319 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:17:48.967206 24319 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:17:48.967242 24319 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:17:48.973419 24319 rpc_server.cc:307] RPC server started. Bound to: 127.20.178.65:46533
I20260929 03:17:48.973471 24447 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.178.65:46533 every 8 connection(s)
I20260929 03:17:48.973788 24319 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651852476208-21193-0/minicluster-data/ts-0/data/info.pb
Sep 29 03:17:48 dist-test-slave-c9qh krb5kdc[24212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651868, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.65@KRBTEST.COM for kudu/127.20.178.126@KRBTEST.COM
I20260929 03:17:48.981359 21193 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuoaHbX/build/release/bin/kudu as pid 24319
I20260929 03:17:48.981462 21193 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651852476208-21193-0/minicluster-data/ts-0/wal/instance
I20260929 03:17:48.981741 21193 external_mini_cluster.cc:1428] Setting key 76a3ea5fd5a7d0cc6038dd11b9298504
I20260929 03:17:48.983675 24297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:48.974511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.65:35755 (local address 127.20.178.126:40105)
0929 03:17:48.974599 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:48.974603 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:48.974792 (+ 189us) server_negotiation.cc:407] Connection header received
0929 03:17:48.975333 (+ 541us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:48.975334 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:48.975347 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:48.975400 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:17:48.976359 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:48.976933 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:48.977971 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:48.978175 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:48.978259 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:48.980347 (+ 2088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:17:48.980349 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:17:48.980349 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:17:48.980357 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:17:48.982207 (+ 1850us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:48.982685 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:48.982686 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:48.982687 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:48.982760 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:48.983012 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:48.983013 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:48.983014 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:48.983133 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:17:48.983206 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 03:17:48.983492 (+ 286us) server_negotiation.cc:299] Negotiation successful
0929 03:17:48.983587 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260929 03:17:48.983865 24450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:48.974489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:40105 (local address 127.20.178.65:35755)
0929 03:17:48.974665 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0929 03:17:48.974709 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 03:17:48.975226 (+ 517us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:17:48.975482 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:17:48.975486 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:17:48.976054 (+ 568us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:17:48.976254 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:48.976258 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:48.977042 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:48.977044 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:48.977862 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:48.977864 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:48.978265 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:48.978266 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:48.978403 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:48.978861 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:17:48.978870 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:17:48.980249 (+ 1379us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:17:48.982281 (+ 2032us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:17:48.982288 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:17:48.982290 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:17:48.982551 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:17:48.982831 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:17:48.982832 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:17:48.982833 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:17:48.982944 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:17:48.983220 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:17:48.983226 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:17:48.983360 (+ 134us) client_negotiation.cc:769] Sending connection context
0929 03:17:48.983460 (+ 100us) client_negotiation.cc:240] Negotiation successful
0929 03:17:48.983619 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":78,"threads_started":1}
I20260929 03:17:48.984354 24448 heartbeater.cc:347] Connected to a master server at 127.20.178.126:40105
I20260929 03:17:48.984437 24448 heartbeater.cc:464] Registering TS with master...
I20260929 03:17:48.984681 24448 heartbeater.cc:511] Master 127.20.178.126:40105 requested a full tablet report, sending...
I20260929 03:17:48.985267 24257 ts_manager.cc:194] Registered new tserver with Master: 2b48d954c83c4ffbb3b7e795d841089b (127.20.178.65:46533)
I20260929 03:17:48.985905 24257 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.178.65@KRBTEST.COM'} at 127.20.178.65:35755
WARNING: no policy specified for kudu/127.20.178.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.178.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.178.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.66.keytab.
Entry for principal kudu/127.20.178.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.66.keytab.
I20260929 03:17:49.008239 21193 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuoaHbX/build/release/bin/kudu
/tmp/dist-test-taskuoaHbX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651852476208-21193-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651852476208-21193-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651852476208-21193-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651852476208-21193-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.178.66:0
--local_ip_for_outbound_sockets=127.20.178.66
--webserver_interface=127.20.178.66
--webserver_port=0
--tserver_master_addrs=127.20.178.126:40105
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.66.keytab
--principal=kudu/127.20.178.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.178.84:43339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651852476208-21193-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651852476208-21193-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651852476208-21193-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:17:49.093387 24453 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:17:49.093582 24453 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:17:49.093603 24453 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:17:49.094666 24453 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:17:49.094699 24453 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:17:49.094724 24453 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.178.66
I20260929 03:17:49.096061 24453 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.178.84:43339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651852476208-21193-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651852476208-21193-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.66.keytab
--principal=kudu/127.20.178.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.178.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651852476208-21193-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.178.66
--webserver_port=0
--tserver_master_addrs=127.20.178.126:40105
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24453
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.178.66
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651852476208-21193-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:17:49.096242 24453 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:17:49.096418 24453 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:17:49.101176 24459 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:17:49.101186 24461 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:17:49.101306 24453 server_base.cc:1034] running on GCE node
W20260929 03:17:49.101262 24458 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:17:49.101621 24453 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:17:49.101879 24453 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:17:49.103018 24453 hybrid_clock.cc:648] HybridClock initialized: now 1790651869103000 us; error 36 us; skew 500 ppm
Sep 29 03:17:49 dist-test-slave-c9qh krb5kdc[24212](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651869, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:17:49.105564 24453 init.cc:377] Logged in from keytab as kudu/127.20.178.66@KRBTEST.COM (short username kudu)
I20260929 03:17:49.105988 24453 webserver.cc:460] Webserver started at http://127.20.178.66:36687/ using document root <none> and password file <none>
I20260929 03:17:49.106235 24453 fs_manager.cc:359] Metadata directory not provided
I20260929 03:17:49.106279 24453 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:17:49.106442 24453 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:17:49.107445 24453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651852476208-21193-0/minicluster-data/ts-1/data/instance:
uuid: "08d42313912249abb98f7fb3c33f9cf2"
format_stamp: "Formatted at 2026-09-29 03:17:49 on dist-test-slave-c9qh"
server_key: "4700e53cd874065c22fecd7a7be5b0ab"
server_key_iv: "7de8b2005d659a95d1c3f1d108991145"
server_key_version: "encryptionkey@0"
I20260929 03:17:49.107805 24453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651852476208-21193-0/minicluster-data/ts-1/wal/instance:
uuid: "08d42313912249abb98f7fb3c33f9cf2"
format_stamp: "Formatted at 2026-09-29 03:17:49 on dist-test-slave-c9qh"
server_key: "4700e53cd874065c22fecd7a7be5b0ab"
server_key_iv: "7de8b2005d659a95d1c3f1d108991145"
server_key_version: "encryptionkey@0"
I20260929 03:17:49.109249 24453 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 03:17:49.109963 24468 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:17:49.110141 24453 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 03:17:49.110232 24453 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651852476208-21193-0/minicluster-data/ts-1/data,/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651852476208-21193-0/minicluster-data/ts-1/wal
uuid: "08d42313912249abb98f7fb3c33f9cf2"
format_stamp: "Formatted at 2026-09-29 03:17:49 on dist-test-slave-c9qh"
server_key: "4700e53cd874065c22fecd7a7be5b0ab"
server_key_iv: "7de8b2005d659a95d1c3f1d108991145"
server_key_version: "encryptionkey@0"
I20260929 03:17:49.110301 24453 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651852476208-21193-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651852476208-21193-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651852476208-21193-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:17:49.133390 24453 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:17:49.135730 24453 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:17:49.135891 24453 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:17:49.136127 24453 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:17:49.136521 24453 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:17:49.136559 24453 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:17:49.136580 24453 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:17:49.136593 24453 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:17:49.142313 24453 rpc_server.cc:307] RPC server started. Bound to: 127.20.178.66:41069
I20260929 03:17:49.142392 24581 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.178.66:41069 every 8 connection(s)
I20260929 03:17:49.142704 24453 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651852476208-21193-0/minicluster-data/ts-1/data/info.pb
I20260929 03:17:49.143466 21193 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuoaHbX/build/release/bin/kudu as pid 24453
I20260929 03:17:49.143567 21193 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651852476208-21193-0/minicluster-data/ts-1/wal/instance
I20260929 03:17:49.143754 21193 external_mini_cluster.cc:1428] Setting key 6d2acf16f25e2c7608d4e75051cf9a81
Sep 29 03:17:49 dist-test-slave-c9qh krb5kdc[24212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651869, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.66@KRBTEST.COM for kudu/127.20.178.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.20.178.67@KRBTEST.COM; defaulting to no policy
I20260929 03:17:49.152144 24297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:49.143450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.66:52523 (local address 127.20.178.126:40105)
0929 03:17:49.143501 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:49.143504 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:49.143699 (+ 195us) server_negotiation.cc:407] Connection header received
0929 03:17:49.144354 (+ 655us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:49.144355 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:49.144366 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:49.144409 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:17:49.145418 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:49.145916 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:49.146922 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:49.147133 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:49.147218 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:49.149334 (+ 2116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:17:49.149335 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:17:49.149335 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:17:49.149344 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:17:49.150756 (+ 1412us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:49.151160 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:49.151161 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:49.151162 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:49.151217 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:49.151505 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:49.151506 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:49.151507 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:49.151618 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:17:49.151689 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0929 03:17:49.151977 (+ 288us) server_negotiation.cc:299] Negotiation successful
0929 03:17:49.152057 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260929 03:17:49.152360 24584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:49.143415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:40105 (local address 127.20.178.66:52523)
0929 03:17:49.143577 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0929 03:17:49.143619 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 03:17:49.144265 (+ 646us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:17:49.144478 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:17:49.144482 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:17:49.145127 (+ 645us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:17:49.145316 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:49.145321 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:49.146009 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:49.146011 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:49.146823 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:49.146825 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:49.147230 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:49.147231 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:49.147356 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:49.147859 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:17:49.147867 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:17:49.149235 (+ 1368us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:17:49.150846 (+ 1611us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:17:49.150852 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:17:49.150855 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:17:49.151073 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:17:49.151308 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:17:49.151309 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:17:49.151310 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:17:49.151443 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:17:49.151690 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:17:49.151697 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:17:49.151839 (+ 142us) client_negotiation.cc:769] Sending connection context
0929 03:17:49.151944 (+ 105us) client_negotiation.cc:240] Negotiation successful
0929 03:17:49.152098 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":91,"threads_started":1}
I20260929 03:17:49.152884 24582 heartbeater.cc:347] Connected to a master server at 127.20.178.126:40105
I20260929 03:17:49.152984 24582 heartbeater.cc:464] Registering TS with master...
I20260929 03:17:49.153151 24582 heartbeater.cc:511] Master 127.20.178.126:40105 requested a full tablet report, sending...
I20260929 03:17:49.153617 24257 ts_manager.cc:194] Registered new tserver with Master: 08d42313912249abb98f7fb3c33f9cf2 (127.20.178.66:41069)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.178.67@KRBTEST.COM" created.
I20260929 03:17:49.154047 24257 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.178.66@KRBTEST.COM'} at 127.20.178.66:52523
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.178.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.67.keytab.
Entry for principal kudu/127.20.178.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.67.keytab.
I20260929 03:17:49.167820 21193 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuoaHbX/build/release/bin/kudu
/tmp/dist-test-taskuoaHbX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651852476208-21193-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651852476208-21193-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651852476208-21193-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651852476208-21193-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.178.67:0
--local_ip_for_outbound_sockets=127.20.178.67
--webserver_interface=127.20.178.67
--webserver_port=0
--tserver_master_addrs=127.20.178.126:40105
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.67.keytab
--principal=kudu/127.20.178.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.178.84:43339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651852476208-21193-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651852476208-21193-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651852476208-21193-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:17:49.247740 24587 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:17:49.247967 24587 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:17:49.248006 24587 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:17:49.249194 24587 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:17:49.249261 24587 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:17:49.249305 24587 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.178.67
I20260929 03:17:49.250752 24587 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.178.84:43339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651852476208-21193-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651852476208-21193-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.67.keytab
--principal=kudu/127.20.178.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.178.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651852476208-21193-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.178.67
--webserver_port=0
--tserver_master_addrs=127.20.178.126:40105
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24587
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.178.67
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651852476208-21193-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:17:49.251017 24587 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:17:49.251242 24587 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:17:49.256089 24595 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:17:49.256196 24592 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:17:49.256079 24593 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:17:49.256348 24587 server_base.cc:1034] running on GCE node
I20260929 03:17:49.256601 24587 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:17:49.256892 24587 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:17:49.258055 24587 hybrid_clock.cc:648] HybridClock initialized: now 1790651869258043 us; error 28 us; skew 500 ppm
Sep 29 03:17:49 dist-test-slave-c9qh krb5kdc[24212](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651869, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:17:49.260326 24587 init.cc:377] Logged in from keytab as kudu/127.20.178.67@KRBTEST.COM (short username kudu)
I20260929 03:17:49.260764 24587 webserver.cc:460] Webserver started at http://127.20.178.67:43357/ using document root <none> and password file <none>
I20260929 03:17:49.260973 24587 fs_manager.cc:359] Metadata directory not provided
I20260929 03:17:49.261016 24587 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:17:49.261102 24587 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:17:49.262115 24587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651852476208-21193-0/minicluster-data/ts-2/data/instance:
uuid: "f154d2c99b474add9340af20873e7ebe"
format_stamp: "Formatted at 2026-09-29 03:17:49 on dist-test-slave-c9qh"
server_key: "2ce6a669c9b23be86cf71eb96a6f2bab"
server_key_iv: "e04f6b9ef59e7e4b9bc9254ba456bf34"
server_key_version: "encryptionkey@0"
I20260929 03:17:49.262405 24587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651852476208-21193-0/minicluster-data/ts-2/wal/instance:
uuid: "f154d2c99b474add9340af20873e7ebe"
format_stamp: "Formatted at 2026-09-29 03:17:49 on dist-test-slave-c9qh"
server_key: "2ce6a669c9b23be86cf71eb96a6f2bab"
server_key_iv: "e04f6b9ef59e7e4b9bc9254ba456bf34"
server_key_version: "encryptionkey@0"
I20260929 03:17:49.263650 24587 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 03:17:49.264286 24602 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:17:49.264439 24587 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 03:17:49.264521 24587 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651852476208-21193-0/minicluster-data/ts-2/data,/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651852476208-21193-0/minicluster-data/ts-2/wal
uuid: "f154d2c99b474add9340af20873e7ebe"
format_stamp: "Formatted at 2026-09-29 03:17:49 on dist-test-slave-c9qh"
server_key: "2ce6a669c9b23be86cf71eb96a6f2bab"
server_key_iv: "e04f6b9ef59e7e4b9bc9254ba456bf34"
server_key_version: "encryptionkey@0"
I20260929 03:17:49.264575 24587 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651852476208-21193-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651852476208-21193-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651852476208-21193-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:17:49.288112 24587 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:17:49.290515 24587 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:17:49.290632 24587 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:17:49.290827 24587 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:17:49.291152 24587 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:17:49.291190 24587 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:17:49.291236 24587 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:17:49.291263 24587 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:17:49.296824 24587 rpc_server.cc:307] RPC server started. Bound to: 127.20.178.67:44847
I20260929 03:17:49.296913 24715 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.178.67:44847 every 8 connection(s)
I20260929 03:17:49.297222 24587 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651852476208-21193-0/minicluster-data/ts-2/data/info.pb
I20260929 03:17:49.302426 21193 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuoaHbX/build/release/bin/kudu as pid 24587
I20260929 03:17:49.302522 21193 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651852476208-21193-0/minicluster-data/ts-2/wal/instance
I20260929 03:17:49.302700 21193 external_mini_cluster.cc:1428] Setting key 06cc8c43e39811c246dd349340450181
Sep 29 03:17:49 dist-test-slave-c9qh krb5kdc[24212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651869, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.67@KRBTEST.COM for kudu/127.20.178.126@KRBTEST.COM
I20260929 03:17:49.306133 24297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:49.298010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.67:38741 (local address 127.20.178.126:40105)
0929 03:17:49.298071 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:49.298073 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:49.298285 (+ 212us) server_negotiation.cc:407] Connection header received
0929 03:17:49.298752 (+ 467us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:49.298753 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:49.298762 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:49.298819 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:17:49.299686 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:49.300166 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:49.301219 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:49.301401 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:49.301456 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:49.303817 (+ 2361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:17:49.303818 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:17:49.303818 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:17:49.303825 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:17:49.305018 (+ 1193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:49.305434 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:49.305435 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:49.305435 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:49.305481 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:49.305687 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:49.305688 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:49.305688 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:49.305788 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:17:49.305832 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0929 03:17:49.306016 (+ 184us) server_negotiation.cc:299] Negotiation successful
0929 03:17:49.306071 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260929 03:17:49.306382 24718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:49.297997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:40105 (local address 127.20.178.67:38741)
0929 03:17:49.298158 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0929 03:17:49.298199 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 03:17:49.298672 (+ 473us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:17:49.298872 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:17:49.298876 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:17:49.299421 (+ 545us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:17:49.299611 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:49.299616 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:49.300255 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:49.300256 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:49.301135 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:49.301137 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:49.301473 (+ 336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:49.301474 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:49.301576 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:49.301995 (+ 419us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:17:49.302003 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:17:49.303727 (+ 1724us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:17:49.305110 (+ 1383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:17:49.305117 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:17:49.305120 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:17:49.305376 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:17:49.305525 (+ 149us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:17:49.305526 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:17:49.305526 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:17:49.305639 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:17:49.305825 (+ 186us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:17:49.305827 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:17:49.305931 (+ 104us) client_negotiation.cc:769] Sending connection context
0929 03:17:49.306007 (+ 76us) client_negotiation.cc:240] Negotiation successful
0929 03:17:49.306119 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":91,"threads_started":1}
I20260929 03:17:49.306918 24716 heartbeater.cc:347] Connected to a master server at 127.20.178.126:40105
I20260929 03:17:49.307025 24716 heartbeater.cc:464] Registering TS with master...
I20260929 03:17:49.307195 24716 heartbeater.cc:511] Master 127.20.178.126:40105 requested a full tablet report, sending...
I20260929 03:17:49.307518 24257 ts_manager.cc:194] Registered new tserver with Master: f154d2c99b474add9340af20873e7ebe (127.20.178.67:44847)
I20260929 03:17:49.307852 24257 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.178.67@KRBTEST.COM'} at 127.20.178.67:38741
I20260929 03:17:49.314472 21193 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 03:17:49 dist-test-slave-c9qh krb5kdc[24212](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651869, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 29 03:17:49 dist-test-slave-c9qh krb5kdc[24212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651869, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.20.178.126@KRBTEST.COM
I20260929 03:17:49.336282 24297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:49.329464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45554 (local address 127.20.178.126:40105)
0929 03:17:49.329533 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:49.329536 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:49.329636 (+ 100us) server_negotiation.cc:407] Connection header received
0929 03:17:49.329729 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:49.329730 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:49.329739 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:49.329806 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:17:49.330404 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:49.330900 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:49.331653 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:49.331848 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:49.331923 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:49.334162 (+ 2239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:17:49.334163 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:17:49.334163 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:17:49.334169 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:17:49.335349 (+ 1180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:49.335648 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:49.335649 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:49.335649 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:49.335698 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:49.335859 (+ 161us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:49.335859 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:49.335859 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:49.335997 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:17:49.336072 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0929 03:17:49.336154 (+ 82us) server_negotiation.cc:299] Negotiation successful
0929 03:17:49.336212 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
W20260929 03:17:49.336546 24257 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:45554
I20260929 03:17:49.337030 21193 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuoaHbX/build/release/bin/kudu with pid 24319
I20260929 03:17:49.341449 21193 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuoaHbX/build/release/bin/kudu with pid 24453
I20260929 03:17:49.345604 21193 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuoaHbX/build/release/bin/kudu with pid 24587
I20260929 03:17:49.349612 21193 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuoaHbX/build/release/bin/kudu with pid 24225
2026-09-29T03:17:49Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (1855 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651852476208-21193-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:17:49 dist-test-slave-c9qh krb5kdc[24728](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:17:49 dist-test-slave-c9qh krb5kdc[24728](info): set up 2 sockets
Sep 29 03:17:49 dist-test-slave-c9qh krb5kdc[24728](info): commencing operation
krb5kdc: starting...
W20260929 03:17:50.388469 21193 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651852476208-21193-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651852476208-21193-0/krb5kdc/test-user.keytab.
Sep 29 03:17:50 dist-test-slave-c9qh krb5kdc[24728](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651870, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:17:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:17:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.178.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.178.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.178.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.126.keytab.
Entry for principal kudu/127.20.178.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.126.keytab.
I20260929 03:17:50.507562 21193 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuoaHbX/build/release/bin/kudu
/tmp/dist-test-taskuoaHbX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651852476208-21193-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651852476208-21193-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651852476208-21193-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651852476208-21193-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.178.126:45721
--webserver_interface=127.20.178.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.126.keytab
--principal=kudu/127.20.178.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.178.84:35763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.178.126:45721
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651852476208-21193-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651852476208-21193-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651852476208-21193-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:17:50.587714 24741 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:17:50.587915 24741 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:17:50.587937 24741 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:17:50.589129 24741 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:17:50.589169 24741 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:17:50.589183 24741 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:17:50.589196 24741 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:17:50.589208 24741 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:17:50.590705 24741 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.178.84:35763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651852476208-21193-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651852476208-21193-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.178.126:45721
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.126.keytab
--principal=kudu/127.20.178.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.178.126:45721
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651852476208-21193-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.178.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24741
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651852476208-21193-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:17:50.590965 24741 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:17:50.591195 24741 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:17:50.596107 24749 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:17:50.596086 24747 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:17:50.596088 24746 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:17:50.596442 24741 server_base.cc:1034] running on GCE node
I20260929 03:17:50.596741 24741 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:17:50.597041 24741 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:17:50.598217 24741 hybrid_clock.cc:648] HybridClock initialized: now 1790651870598212 us; error 38 us; skew 500 ppm
Sep 29 03:17:50 dist-test-slave-c9qh krb5kdc[24728](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651870, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:17:50.600767 24741 init.cc:377] Logged in from keytab as kudu/127.20.178.126@KRBTEST.COM (short username kudu)
I20260929 03:17:50.601243 24741 webserver.cc:460] Webserver started at http://127.20.178.126:43863/ using document root <none> and password file <none>
I20260929 03:17:50.601470 24741 fs_manager.cc:359] Metadata directory not provided
I20260929 03:17:50.601536 24741 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:17:50.601686 24741 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:17:50.602782 24741 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651852476208-21193-0/minicluster-data/master-0/data/instance:
uuid: "b6e3d97993ef4c1faf5ff00e3b277f2a"
format_stamp: "Formatted at 2026-09-29 03:17:50 on dist-test-slave-c9qh"
server_key: "89d6ae9fd905f43254135ca4a2a49ab4"
server_key_iv: "874d23098f2ddb78b946f36026dc3b1e"
server_key_version: "encryptionkey@0"
I20260929 03:17:50.603185 24741 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651852476208-21193-0/minicluster-data/master-0/wal/instance:
uuid: "b6e3d97993ef4c1faf5ff00e3b277f2a"
format_stamp: "Formatted at 2026-09-29 03:17:50 on dist-test-slave-c9qh"
server_key: "89d6ae9fd905f43254135ca4a2a49ab4"
server_key_iv: "874d23098f2ddb78b946f36026dc3b1e"
server_key_version: "encryptionkey@0"
I20260929 03:17:50.604553 24741 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260929 03:17:50.605330 24756 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:17:50.605499 24741 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 03:17:50.605598 24741 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651852476208-21193-0/minicluster-data/master-0/data,/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651852476208-21193-0/minicluster-data/master-0/wal
uuid: "b6e3d97993ef4c1faf5ff00e3b277f2a"
format_stamp: "Formatted at 2026-09-29 03:17:50 on dist-test-slave-c9qh"
server_key: "89d6ae9fd905f43254135ca4a2a49ab4"
server_key_iv: "874d23098f2ddb78b946f36026dc3b1e"
server_key_version: "encryptionkey@0"
I20260929 03:17:50.605671 24741 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651852476208-21193-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651852476208-21193-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651852476208-21193-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:17:50.618660 24741 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:17:50.621129 24741 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:17:50.621305 24741 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:17:50.625267 24741 rpc_server.cc:307] RPC server started. Bound to: 127.20.178.126:45721
I20260929 03:17:50.625322 24808 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.178.126:45721 every 8 connection(s)
I20260929 03:17:50.625833 24741 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651852476208-21193-0/minicluster-data/master-0/data/info.pb
I20260929 03:17:50.626485 24809 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:17:50.628913 24809 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6e3d97993ef4c1faf5ff00e3b277f2a: Bootstrap starting.
I20260929 03:17:50.629755 24809 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b6e3d97993ef4c1faf5ff00e3b277f2a: Neither blocks nor log segments found. Creating new log.
I20260929 03:17:50.630119 24809 log.cc:824] T 00000000000000000000000000000000 P b6e3d97993ef4c1faf5ff00e3b277f2a: Log is configured to *not* fsync() on all Append() calls
I20260929 03:17:50.630769 24809 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6e3d97993ef4c1faf5ff00e3b277f2a: No bootstrap required, opened a new log
I20260929 03:17:50.632141 21193 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuoaHbX/build/release/bin/kudu as pid 24741
I20260929 03:17:50.632257 21193 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651852476208-21193-0/minicluster-data/master-0/wal/instance
I20260929 03:17:50.632448 21193 external_mini_cluster.cc:1428] Setting key a3fc84b5f32fde187e39768e888eb09e
I20260929 03:17:50.632402 24809 raft_consensus.cc:348] T 00000000000000000000000000000000 P b6e3d97993ef4c1faf5ff00e3b277f2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6e3d97993ef4c1faf5ff00e3b277f2a" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 45721 } }
I20260929 03:17:50.632611 24809 raft_consensus.cc:374] T 00000000000000000000000000000000 P b6e3d97993ef4c1faf5ff00e3b277f2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:17:50.632709 24809 raft_consensus.cc:729] T 00000000000000000000000000000000 P b6e3d97993ef4c1faf5ff00e3b277f2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6e3d97993ef4c1faf5ff00e3b277f2a, State: Initialized, Role: FOLLOWER
I20260929 03:17:50.632872 24809 consensus_queue.cc:260] T 00000000000000000000000000000000 P b6e3d97993ef4c1faf5ff00e3b277f2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6e3d97993ef4c1faf5ff00e3b277f2a" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 45721 } }
I20260929 03:17:50.632968 24809 raft_consensus.cc:388] T 00000000000000000000000000000000 P b6e3d97993ef4c1faf5ff00e3b277f2a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:17:50.633011 24809 raft_consensus.cc:482] T 00000000000000000000000000000000 P b6e3d97993ef4c1faf5ff00e3b277f2a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:17:50.633061 24809 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b6e3d97993ef4c1faf5ff00e3b277f2a [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:17:50.633653 24809 raft_consensus.cc:504] T 00000000000000000000000000000000 P b6e3d97993ef4c1faf5ff00e3b277f2a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6e3d97993ef4c1faf5ff00e3b277f2a" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 45721 } }
I20260929 03:17:50.633744 24809 leader_election.cc:302] T 00000000000000000000000000000000 P b6e3d97993ef4c1faf5ff00e3b277f2a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b6e3d97993ef4c1faf5ff00e3b277f2a; no voters:
I20260929 03:17:50.633939 24809 leader_election.cc:288] T 00000000000000000000000000000000 P b6e3d97993ef4c1faf5ff00e3b277f2a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:17:50.633968 24814 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b6e3d97993ef4c1faf5ff00e3b277f2a [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:17:50.634222 24809 sys_catalog.cc:564] T 00000000000000000000000000000000 P b6e3d97993ef4c1faf5ff00e3b277f2a [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:17:50.634229 24814 raft_consensus.cc:686] T 00000000000000000000000000000000 P b6e3d97993ef4c1faf5ff00e3b277f2a [term 1 LEADER]: Becoming Leader. State: Replica: b6e3d97993ef4c1faf5ff00e3b277f2a, State: Running, Role: LEADER
I20260929 03:17:50.634474 24814 consensus_queue.cc:237] T 00000000000000000000000000000000 P b6e3d97993ef4c1faf5ff00e3b277f2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6e3d97993ef4c1faf5ff00e3b277f2a" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 45721 } }
I20260929 03:17:50.634866 24815 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6e3d97993ef4c1faf5ff00e3b277f2a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b6e3d97993ef4c1faf5ff00e3b277f2a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6e3d97993ef4c1faf5ff00e3b277f2a" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 45721 } } }
I20260929 03:17:50.634953 24815 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6e3d97993ef4c1faf5ff00e3b277f2a [sys.catalog]: This master's current role is: LEADER
I20260929 03:17:50.635080 24816 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6e3d97993ef4c1faf5ff00e3b277f2a [sys.catalog]: SysCatalogTable state changed. Reason: New leader b6e3d97993ef4c1faf5ff00e3b277f2a. Latest consensus state: current_term: 1 leader_uuid: "b6e3d97993ef4c1faf5ff00e3b277f2a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6e3d97993ef4c1faf5ff00e3b277f2a" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 45721 } } }
I20260929 03:17:50.635124 24816 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6e3d97993ef4c1faf5ff00e3b277f2a [sys.catalog]: This master's current role is: LEADER
I20260929 03:17:50.635226 24823 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:17:50.635710 24823 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:17:50.638025 24823 catalog_manager.cc:1347] Generated new cluster ID: 68f3647075204daf8f023f5021a52d20
I20260929 03:17:50.638084 24823 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 29 03:17:50 dist-test-slave-c9qh krb5kdc[24728](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651870, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.178.126@KRBTEST.COM
I20260929 03:17:50.643545 24823 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:17:50.643580 24813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:50.632997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40736 (local address 127.20.178.126:45721)
0929 03:17:50.633554 (+ 557us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:50.633564 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:50.633604 (+ 40us) server_negotiation.cc:407] Connection header received
0929 03:17:50.633705 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:50.633732 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:50.633789 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:50.633959 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:17:50.634736 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:50.636150 (+ 1414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:50.637188 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:50.637789 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:50.637909 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:50.640232 (+ 2323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:17:50.640236 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:17:50.640237 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:17:50.640249 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:17:50.641983 (+ 1734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:50.642357 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:50.642360 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:50.642361 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:50.642462 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:50.642680 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:50.642681 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:50.642681 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:50.642915 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:17:50.643084 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0929 03:17:50.643182 (+ 98us) server_negotiation.cc:299] Negotiation successful
0929 03:17:50.643322 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":506,"thread_start_us":477,"threads_started":1}
I20260929 03:17:50.644222 24823 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:17:50.653216 24823 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b6e3d97993ef4c1faf5ff00e3b277f2a: Generated new TSK 0
I20260929 03:17:50.653400 24823 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.178.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.178.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.178.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.65.keytab.
Entry for principal kudu/127.20.178.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.65.keytab.
I20260929 03:17:50.681880 21193 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuoaHbX/build/release/bin/kudu
/tmp/dist-test-taskuoaHbX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651852476208-21193-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651852476208-21193-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651852476208-21193-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651852476208-21193-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.178.65:0
--local_ip_for_outbound_sockets=127.20.178.65
--webserver_interface=127.20.178.65
--webserver_port=0
--tserver_master_addrs=127.20.178.126:45721
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.65.keytab
--principal=kudu/127.20.178.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.178.84:35763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651852476208-21193-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651852476208-21193-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651852476208-21193-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:17:50.782297 24835 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:17:50.782596 24835 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:17:50.782631 24835 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:17:50.783798 24835 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:17:50.783840 24835 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:17:50.783866 24835 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.178.65
I20260929 03:17:50.785363 24835 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.178.84:35763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651852476208-21193-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651852476208-21193-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.65.keytab
--principal=kudu/127.20.178.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.178.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651852476208-21193-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.178.65
--webserver_port=0
--tserver_master_addrs=127.20.178.126:45721
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24835
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.178.65
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651852476208-21193-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:17:50.785607 24835 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:17:50.785844 24835 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:17:50.792078 24843 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:17:50.792078 24841 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:17:50.792485 24835 server_base.cc:1034] running on GCE node
W20260929 03:17:50.792116 24840 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:17:50.792781 24835 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:17:50.793026 24835 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:17:50.794198 24835 hybrid_clock.cc:648] HybridClock initialized: now 1790651870794184 us; error 31 us; skew 500 ppm
Sep 29 03:17:50 dist-test-slave-c9qh krb5kdc[24728](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651870, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:17:50.796738 24835 init.cc:377] Logged in from keytab as kudu/127.20.178.65@KRBTEST.COM (short username kudu)
I20260929 03:17:50.797247 24835 webserver.cc:460] Webserver started at http://127.20.178.65:41827/ using document root <none> and password file <none>
I20260929 03:17:50.797475 24835 fs_manager.cc:359] Metadata directory not provided
I20260929 03:17:50.797542 24835 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:17:50.797683 24835 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:17:50.798734 24835 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651852476208-21193-0/minicluster-data/ts-0/data/instance:
uuid: "20e215b778ba440e8a2842190b7ea449"
format_stamp: "Formatted at 2026-09-29 03:17:50 on dist-test-slave-c9qh"
server_key: "a17d203ef1ad4ff57933f8e608242743"
server_key_iv: "c35db37d2b03d4e58d5dd45e9c370a9b"
server_key_version: "encryptionkey@0"
I20260929 03:17:50.799096 24835 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651852476208-21193-0/minicluster-data/ts-0/wal/instance:
uuid: "20e215b778ba440e8a2842190b7ea449"
format_stamp: "Formatted at 2026-09-29 03:17:50 on dist-test-slave-c9qh"
server_key: "a17d203ef1ad4ff57933f8e608242743"
server_key_iv: "c35db37d2b03d4e58d5dd45e9c370a9b"
server_key_version: "encryptionkey@0"
I20260929 03:17:50.800408 24835 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 03:17:50.801235 24850 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:17:50.801424 24835 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260929 03:17:50.801520 24835 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651852476208-21193-0/minicluster-data/ts-0/data,/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651852476208-21193-0/minicluster-data/ts-0/wal
uuid: "20e215b778ba440e8a2842190b7ea449"
format_stamp: "Formatted at 2026-09-29 03:17:50 on dist-test-slave-c9qh"
server_key: "a17d203ef1ad4ff57933f8e608242743"
server_key_iv: "c35db37d2b03d4e58d5dd45e9c370a9b"
server_key_version: "encryptionkey@0"
I20260929 03:17:50.801609 24835 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651852476208-21193-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651852476208-21193-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651852476208-21193-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:17:50.813719 24835 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:17:50.815974 24835 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:17:50.816134 24835 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:17:50.816371 24835 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:17:50.816718 24835 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:17:50.816779 24835 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:17:50.816820 24835 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:17:50.816848 24835 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:17:50.822407 24835 rpc_server.cc:307] RPC server started. Bound to: 127.20.178.65:38087
I20260929 03:17:50.822480 24963 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.178.65:38087 every 8 connection(s)
I20260929 03:17:50.822801 24835 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651852476208-21193-0/minicluster-data/ts-0/data/info.pb
I20260929 03:17:50.827430 21193 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuoaHbX/build/release/bin/kudu as pid 24835
I20260929 03:17:50.827550 21193 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651852476208-21193-0/minicluster-data/ts-0/wal/instance
I20260929 03:17:50.827729 21193 external_mini_cluster.cc:1428] Setting key 8b570a14db8765df5319d2cc220e0d69
Sep 29 03:17:50 dist-test-slave-c9qh krb5kdc[24728](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651870, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.65@KRBTEST.COM for kudu/127.20.178.126@KRBTEST.COM
I20260929 03:17:50.832966 24813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:50.823546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.65:34847 (local address 127.20.178.126:45721)
0929 03:17:50.823624 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:50.823627 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:50.823837 (+ 210us) server_negotiation.cc:407] Connection header received
0929 03:17:50.824801 (+ 964us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:50.824802 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:50.824815 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:50.824867 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:17:50.825413 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:50.825901 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:50.826836 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:50.827022 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:50.827072 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:50.829288 (+ 2216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:17:50.829290 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:17:50.829291 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:17:50.829300 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:17:50.831031 (+ 1731us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:50.831503 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:50.831504 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:50.831505 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:50.831610 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:50.831952 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:50.831953 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:50.831953 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:50.832194 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:17:50.832284 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0929 03:17:50.832701 (+ 417us) server_negotiation.cc:299] Negotiation successful
0929 03:17:50.832845 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 03:17:50.833240 24966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:50.823509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:45721 (local address 127.20.178.65:34847)
0929 03:17:50.823717 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0929 03:17:50.823754 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 03:17:50.824626 (+ 872us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:17:50.824921 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:17:50.824924 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:17:50.825162 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:17:50.825345 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:50.825349 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:50.825982 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:50.825982 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:50.826764 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:50.826766 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:50.827092 (+ 326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:50.827092 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:50.827186 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:50.827729 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:17:50.827737 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:17:50.829189 (+ 1452us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:17:50.831121 (+ 1932us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:17:50.831128 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:17:50.831130 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:17:50.831417 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:17:50.831712 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:17:50.831713 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:17:50.831714 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:17:50.831856 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:17:50.832316 (+ 460us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:17:50.832324 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:17:50.832491 (+ 167us) client_negotiation.cc:769] Sending connection context
0929 03:17:50.832663 (+ 172us) client_negotiation.cc:240] Negotiation successful
0929 03:17:50.832843 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":104,"threads_started":1}
I20260929 03:17:50.834731 24964 heartbeater.cc:347] Connected to a master server at 127.20.178.126:45721
I20260929 03:17:50.834824 24964 heartbeater.cc:464] Registering TS with master...
I20260929 03:17:50.835173 24964 heartbeater.cc:511] Master 127.20.178.126:45721 requested a full tablet report, sending...
I20260929 03:17:50.835744 24773 ts_manager.cc:194] Registered new tserver with Master: 20e215b778ba440e8a2842190b7ea449 (127.20.178.65:38087)
I20260929 03:17:50.836375 24773 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.178.65@KRBTEST.COM'} at 127.20.178.65:34847
WARNING: no policy specified for kudu/127.20.178.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.178.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.178.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.66.keytab.
Entry for principal kudu/127.20.178.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.66.keytab.
I20260929 03:17:50.855046 21193 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuoaHbX/build/release/bin/kudu
/tmp/dist-test-taskuoaHbX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651852476208-21193-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651852476208-21193-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651852476208-21193-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651852476208-21193-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.178.66:0
--local_ip_for_outbound_sockets=127.20.178.66
--webserver_interface=127.20.178.66
--webserver_port=0
--tserver_master_addrs=127.20.178.126:45721
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.66.keytab
--principal=kudu/127.20.178.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.178.84:35763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651852476208-21193-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651852476208-21193-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651852476208-21193-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:17:50.936268 24969 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:17:50.936480 24969 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:17:50.936504 24969 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:17:50.937624 24969 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:17:50.937662 24969 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:17:50.937688 24969 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.178.66
I20260929 03:17:50.939124 24969 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.178.84:35763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651852476208-21193-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651852476208-21193-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.66.keytab
--principal=kudu/127.20.178.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.178.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651852476208-21193-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.178.66
--webserver_port=0
--tserver_master_addrs=127.20.178.126:45721
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24969
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.178.66
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651852476208-21193-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:17:50.939319 24969 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:17:50.939498 24969 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:17:50.944474 24975 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:17:50.944489 24977 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:17:50.944497 24974 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:17:50.944717 24969 server_base.cc:1034] running on GCE node
I20260929 03:17:50.945008 24969 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:17:50.945216 24969 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:17:50.946359 24969 hybrid_clock.cc:648] HybridClock initialized: now 1790651870946349 us; error 26 us; skew 500 ppm
Sep 29 03:17:50 dist-test-slave-c9qh krb5kdc[24728](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651870, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:17:50.948930 24969 init.cc:377] Logged in from keytab as kudu/127.20.178.66@KRBTEST.COM (short username kudu)
I20260929 03:17:50.949426 24969 webserver.cc:460] Webserver started at http://127.20.178.66:41407/ using document root <none> and password file <none>
I20260929 03:17:50.949677 24969 fs_manager.cc:359] Metadata directory not provided
I20260929 03:17:50.949724 24969 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:17:50.949856 24969 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:17:50.950901 24969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651852476208-21193-0/minicluster-data/ts-1/data/instance:
uuid: "9c586b8bda764e2192c204f4074d65ad"
format_stamp: "Formatted at 2026-09-29 03:17:50 on dist-test-slave-c9qh"
server_key: "1621f528e22d7152d0afdd4aefa81369"
server_key_iv: "dc2f321062092a59bbeb5d1481b2acd5"
server_key_version: "encryptionkey@0"
I20260929 03:17:50.951246 24969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651852476208-21193-0/minicluster-data/ts-1/wal/instance:
uuid: "9c586b8bda764e2192c204f4074d65ad"
format_stamp: "Formatted at 2026-09-29 03:17:50 on dist-test-slave-c9qh"
server_key: "1621f528e22d7152d0afdd4aefa81369"
server_key_iv: "dc2f321062092a59bbeb5d1481b2acd5"
server_key_version: "encryptionkey@0"
I20260929 03:17:50.952603 24969 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 03:17:50.953405 24984 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:17:50.953598 24969 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 03:17:50.953691 24969 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651852476208-21193-0/minicluster-data/ts-1/data,/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651852476208-21193-0/minicluster-data/ts-1/wal
uuid: "9c586b8bda764e2192c204f4074d65ad"
format_stamp: "Formatted at 2026-09-29 03:17:50 on dist-test-slave-c9qh"
server_key: "1621f528e22d7152d0afdd4aefa81369"
server_key_iv: "dc2f321062092a59bbeb5d1481b2acd5"
server_key_version: "encryptionkey@0"
I20260929 03:17:50.953743 24969 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651852476208-21193-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651852476208-21193-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651852476208-21193-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:17:50.975767 24969 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:17:50.977748 24969 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:17:50.977864 24969 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:17:50.978055 24969 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:17:50.978370 24969 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:17:50.978410 24969 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:17:50.978432 24969 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:17:50.978478 24969 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:17:50.984143 24969 rpc_server.cc:307] RPC server started. Bound to: 127.20.178.66:38811
I20260929 03:17:50.984205 25097 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.178.66:38811 every 8 connection(s)
I20260929 03:17:50.984539 24969 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651852476208-21193-0/minicluster-data/ts-1/data/info.pb
I20260929 03:17:50.989813 21193 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuoaHbX/build/release/bin/kudu as pid 24969
I20260929 03:17:50.989933 21193 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651852476208-21193-0/minicluster-data/ts-1/wal/instance
I20260929 03:17:50.990098 21193 external_mini_cluster.cc:1428] Setting key 3c0bdf02c8075b78fa85f760c5823943
Sep 29 03:17:50 dist-test-slave-c9qh krb5kdc[24728](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651870, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.66@KRBTEST.COM for kudu/127.20.178.126@KRBTEST.COM
I20260929 03:17:50.995117 24813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:50.985312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.66:60913 (local address 127.20.178.126:45721)
0929 03:17:50.985379 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:50.985381 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:50.985589 (+ 208us) server_negotiation.cc:407] Connection header received
0929 03:17:50.986484 (+ 895us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:50.986485 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:50.986495 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:50.986596 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:17:50.987185 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:50.987813 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:50.988789 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:50.988977 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:50.989036 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:50.991515 (+ 2479us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:17:50.991517 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:17:50.991517 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:17:50.991523 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:17:50.993657 (+ 2134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:50.994151 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:50.994152 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:50.994152 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:50.994204 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:50.994483 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:50.994484 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:50.994484 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:50.994581 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:17:50.994631 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0929 03:17:50.994984 (+ 353us) server_negotiation.cc:299] Negotiation successful
0929 03:17:50.995053 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260929 03:17:50.995404 25100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:50.985285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:45721 (local address 127.20.178.66:60913)
0929 03:17:50.985463 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0929 03:17:50.985505 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 03:17:50.986397 (+ 892us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:17:50.986674 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:17:50.986677 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:17:50.986923 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:17:50.987109 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:50.987113 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:50.987897 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:50.987898 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:50.988715 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:50.988717 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:50.989050 (+ 333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:50.989051 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:50.989151 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:50.989675 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:17:50.989687 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:17:50.991402 (+ 1715us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:17:50.993739 (+ 2337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:17:50.993746 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:17:50.993749 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:17:50.994071 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:17:50.994272 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:17:50.994273 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:17:50.994273 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:17:50.994418 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:17:50.994646 (+ 228us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:17:50.994652 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:17:50.994836 (+ 184us) client_negotiation.cc:769] Sending connection context
0929 03:17:50.994971 (+ 135us) client_negotiation.cc:240] Negotiation successful
0929 03:17:50.995152 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":105,"threads_started":1}
I20260929 03:17:50.995934 25098 heartbeater.cc:347] Connected to a master server at 127.20.178.126:45721
I20260929 03:17:50.996048 25098 heartbeater.cc:464] Registering TS with master...
I20260929 03:17:50.996274 25098 heartbeater.cc:511] Master 127.20.178.126:45721 requested a full tablet report, sending...
I20260929 03:17:50.996724 24773 ts_manager.cc:194] Registered new tserver with Master: 9c586b8bda764e2192c204f4074d65ad (127.20.178.66:38811)
I20260929 03:17:50.997108 24773 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.178.66@KRBTEST.COM'} at 127.20.178.66:60913
WARNING: no policy specified for kudu/127.20.178.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.178.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.178.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.67.keytab.
Entry for principal kudu/127.20.178.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.67.keytab.
I20260929 03:17:51.015705 21193 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuoaHbX/build/release/bin/kudu
/tmp/dist-test-taskuoaHbX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651852476208-21193-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651852476208-21193-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651852476208-21193-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651852476208-21193-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.178.67:0
--local_ip_for_outbound_sockets=127.20.178.67
--webserver_interface=127.20.178.67
--webserver_port=0
--tserver_master_addrs=127.20.178.126:45721
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.67.keytab
--principal=kudu/127.20.178.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.178.84:35763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651852476208-21193-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651852476208-21193-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651852476208-21193-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:17:51.096001 25103 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:17:51.096204 25103 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:17:51.096226 25103 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:17:51.097312 25103 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:17:51.097348 25103 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:17:51.097373 25103 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.178.67
I20260929 03:17:51.098773 25103 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.178.84:35763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651852476208-21193-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651852476208-21193-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.67.keytab
--principal=kudu/127.20.178.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.178.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651852476208-21193-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.178.67
--webserver_port=0
--tserver_master_addrs=127.20.178.126:45721
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25103
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.178.67
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651852476208-21193-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:17:51.098999 25103 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:17:51.099171 25103 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 03:17:51.104148 25103 server_base.cc:1034] running on GCE node
W20260929 03:17:51.104187 25111 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:17:51.104111 25109 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:17:51.104087 25108 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:17:51.104781 25103 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:17:51.105054 25103 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:17:51.106215 25103 hybrid_clock.cc:648] HybridClock initialized: now 1790651871106196 us; error 33 us; skew 500 ppm
Sep 29 03:17:51 dist-test-slave-c9qh krb5kdc[24728](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651871, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:17:51.108788 25103 init.cc:377] Logged in from keytab as kudu/127.20.178.67@KRBTEST.COM (short username kudu)
I20260929 03:17:51.109275 25103 webserver.cc:460] Webserver started at http://127.20.178.67:41815/ using document root <none> and password file <none>
I20260929 03:17:51.109514 25103 fs_manager.cc:359] Metadata directory not provided
I20260929 03:17:51.109587 25103 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:17:51.109740 25103 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:17:51.110745 25103 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651852476208-21193-0/minicluster-data/ts-2/data/instance:
uuid: "b51536dddf954433a2a33381bdf7d2dc"
format_stamp: "Formatted at 2026-09-29 03:17:51 on dist-test-slave-c9qh"
server_key: "7ad18b0811937d46a872f46800aebb83"
server_key_iv: "27f02d0a32a4dd2e5b3f8eb122de5e1a"
server_key_version: "encryptionkey@0"
I20260929 03:17:51.111104 25103 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651852476208-21193-0/minicluster-data/ts-2/wal/instance:
uuid: "b51536dddf954433a2a33381bdf7d2dc"
format_stamp: "Formatted at 2026-09-29 03:17:51 on dist-test-slave-c9qh"
server_key: "7ad18b0811937d46a872f46800aebb83"
server_key_iv: "27f02d0a32a4dd2e5b3f8eb122de5e1a"
server_key_version: "encryptionkey@0"
I20260929 03:17:51.112422 25103 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260929 03:17:51.113277 25118 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:17:51.113451 25103 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 03:17:51.113545 25103 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651852476208-21193-0/minicluster-data/ts-2/data,/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651852476208-21193-0/minicluster-data/ts-2/wal
uuid: "b51536dddf954433a2a33381bdf7d2dc"
format_stamp: "Formatted at 2026-09-29 03:17:51 on dist-test-slave-c9qh"
server_key: "7ad18b0811937d46a872f46800aebb83"
server_key_iv: "27f02d0a32a4dd2e5b3f8eb122de5e1a"
server_key_version: "encryptionkey@0"
I20260929 03:17:51.113668 25103 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651852476208-21193-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651852476208-21193-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651852476208-21193-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:17:51.130025 25103 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:17:51.132005 25103 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:17:51.132164 25103 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:17:51.132373 25103 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:17:51.132758 25103 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:17:51.132810 25103 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:17:51.132853 25103 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:17:51.132881 25103 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:17:51.139073 25103 rpc_server.cc:307] RPC server started. Bound to: 127.20.178.67:37091
I20260929 03:17:51.139135 25231 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.178.67:37091 every 8 connection(s)
I20260929 03:17:51.139468 25103 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651852476208-21193-0/minicluster-data/ts-2/data/info.pb
I20260929 03:17:51.140475 21193 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuoaHbX/build/release/bin/kudu as pid 25103
I20260929 03:17:51.140584 21193 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651852476208-21193-0/minicluster-data/ts-2/wal/instance
I20260929 03:17:51.140816 21193 external_mini_cluster.cc:1428] Setting key 50fba1223bb9576c8258de422a8491a9
Sep 29 03:17:51 dist-test-slave-c9qh krb5kdc[24728](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651871, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.67@KRBTEST.COM for kudu/127.20.178.126@KRBTEST.COM
I20260929 03:17:51.148916 24813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:51.140240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.67:39073 (local address 127.20.178.126:45721)
0929 03:17:51.140338 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:51.140340 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:51.140536 (+ 196us) server_negotiation.cc:407] Connection header received
0929 03:17:51.141770 (+ 1234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:51.141771 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:51.141788 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:51.141824 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:17:51.142454 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:51.142939 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:51.143923 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:51.144106 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:51.144173 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:51.146388 (+ 2215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:17:51.146389 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:17:51.146390 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:17:51.146396 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:17:51.147707 (+ 1311us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:51.148073 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:51.148074 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:51.148074 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:51.148128 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:51.148344 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:51.148344 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:51.148345 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:51.148451 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:17:51.148500 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0929 03:17:51.148778 (+ 278us) server_negotiation.cc:299] Negotiation successful
0929 03:17:51.148842 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 03:17:51.149085 25234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:51.140171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:45721 (local address 127.20.178.67:39073)
0929 03:17:51.140379 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0929 03:17:51.140433 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0929 03:17:51.141652 (+ 1219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:17:51.141886 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:17:51.141889 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:17:51.142182 (+ 293us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:17:51.142374 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:51.142379 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:51.143037 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:51.143039 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:51.143842 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:51.143844 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:51.144192 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:51.144193 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:51.144292 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:51.144866 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:17:51.144874 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:17:51.146294 (+ 1420us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:17:51.147780 (+ 1486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:17:51.147786 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:17:51.147787 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:17:51.148005 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:17:51.148192 (+ 187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:17:51.148192 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:17:51.148192 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:17:51.148288 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:17:51.148506 (+ 218us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:17:51.148510 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:17:51.148624 (+ 114us) client_negotiation.cc:769] Sending connection context
0929 03:17:51.148759 (+ 135us) client_negotiation.cc:240] Negotiation successful
0929 03:17:51.148878 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":103,"threads_started":1}
I20260929 03:17:51.149540 25232 heartbeater.cc:347] Connected to a master server at 127.20.178.126:45721
I20260929 03:17:51.149636 25232 heartbeater.cc:464] Registering TS with master...
I20260929 03:17:51.149806 25232 heartbeater.cc:511] Master 127.20.178.126:45721 requested a full tablet report, sending...
I20260929 03:17:51.150152 24773 ts_manager.cc:194] Registered new tserver with Master: b51536dddf954433a2a33381bdf7d2dc (127.20.178.67:37091)
I20260929 03:17:51.150483 24773 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.178.67@KRBTEST.COM'} at 127.20.178.67:39073
I20260929 03:17:51.152736 21193 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 03:17:51 dist-test-slave-c9qh krb5kdc[24728](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651871, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 03:17:51 dist-test-slave-c9qh krb5kdc[24728](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651871, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.178.65@KRBTEST.COM
I20260929 03:17:51.175832 25239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:51.168143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33234 (local address 127.20.178.65:38087)
0929 03:17:51.168297 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:51.168301 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:51.168310 (+ 9us) server_negotiation.cc:407] Connection header received
0929 03:17:51.168335 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:51.168337 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:51.168350 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:51.168434 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:17:51.169082 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:51.169820 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:51.170560 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:51.170875 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:51.170963 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:51.173245 (+ 2282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:17:51.173252 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:17:51.173253 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:17:51.173259 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:17:51.174761 (+ 1502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:51.175129 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:51.175132 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:51.175132 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:51.175198 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:51.175367 (+ 169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:51.175367 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:51.175368 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:51.175487 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:17:51.175583 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0929 03:17:51.175692 (+ 109us) server_negotiation.cc:299] Negotiation successful
0929 03:17:51.175752 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":68,"threads_started":1}
Sep 29 03:17:51 dist-test-slave-c9qh krb5kdc[24728](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651871, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.178.126@KRBTEST.COM
I20260929 03:17:51.183267 24813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:51.177165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40748 (local address 127.20.178.126:45721)
0929 03:17:51.177211 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:51.177213 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:51.177225 (+ 12us) server_negotiation.cc:407] Connection header received
0929 03:17:51.177270 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:51.177271 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:51.177279 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:51.177319 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:17:51.177766 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:51.178255 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:51.178911 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:51.179086 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:51.179152 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:51.181242 (+ 2090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:17:51.181244 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:17:51.181244 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:17:51.181250 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:17:51.182378 (+ 1128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:51.182669 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:51.182670 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:51.182670 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:51.182718 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:51.182858 (+ 140us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:51.182859 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:51.182859 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:51.182983 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:17:51.183052 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0929 03:17:51.183144 (+ 92us) server_negotiation.cc:299] Negotiation successful
0929 03:17:51.183203 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
W20260929 03:17:51.183517 24773 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40748
I20260929 03:17:51.183876 21193 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuoaHbX/build/release/bin/kudu with pid 24835
I20260929 03:17:51.188372 21193 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuoaHbX/build/release/bin/kudu with pid 24969
I20260929 03:17:51.192493 21193 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuoaHbX/build/release/bin/kudu with pid 25103
I20260929 03:17:51.196810 21193 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuoaHbX/build/release/bin/kudu with pid 24741
2026-09-29T03:17:51Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (1846 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651852476208-21193-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:17:51 dist-test-slave-c9qh krb5kdc[25244](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:17:51 dist-test-slave-c9qh krb5kdc[25244](info): set up 2 sockets
Sep 29 03:17:51 dist-test-slave-c9qh krb5kdc[25244](info): commencing operation
krb5kdc: starting...
W20260929 03:17:52.225170 21193 mini_kdc.cc:121] Time spent starting KDC: real 1.009s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651852476208-21193-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651852476208-21193-0/krb5kdc/test-user.keytab.
Sep 29 03:17:52 dist-test-slave-c9qh krb5kdc[25244](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651872, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:17:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:17:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.178.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.178.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.178.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.126.keytab.
Entry for principal kudu/127.20.178.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.126.keytab.
I20260929 03:17:52.341455 21193 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuoaHbX/build/release/bin/kudu
/tmp/dist-test-taskuoaHbX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651852476208-21193-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651852476208-21193-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651852476208-21193-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651852476208-21193-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.178.126:44497
--webserver_interface=127.20.178.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.126.keytab
--principal=kudu/127.20.178.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.178.84:44819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.178.126:44497
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651852476208-21193-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651852476208-21193-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651852476208-21193-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:17:52.420092 25257 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:17:52.420310 25257 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:17:52.420333 25257 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:17:52.421712 25257 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:17:52.421762 25257 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:17:52.421777 25257 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:17:52.421789 25257 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:17:52.421837 25257 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:17:52.423278 25257 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.178.84:44819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651852476208-21193-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651852476208-21193-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.178.126:44497
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.126.keytab
--principal=kudu/127.20.178.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.178.126:44497
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651852476208-21193-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.178.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25257
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651852476208-21193-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:17:52.423530 25257 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:17:52.423766 25257 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:17:52.428320 25265 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:17:52.428318 25263 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:17:52.428474 25257 server_base.cc:1034] running on GCE node
W20260929 03:17:52.428339 25262 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:17:52.428819 25257 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:17:52.429095 25257 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:17:52.430262 25257 hybrid_clock.cc:648] HybridClock initialized: now 1790651872430252 us; error 29 us; skew 500 ppm
Sep 29 03:17:52 dist-test-slave-c9qh krb5kdc[25244](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651872, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:17:52.432821 25257 init.cc:377] Logged in from keytab as kudu/127.20.178.126@KRBTEST.COM (short username kudu)
I20260929 03:17:52.433188 25257 fs_manager.cc:359] Metadata directory not provided
I20260929 03:17:52.433236 25257 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:17:52.433398 25257 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:17:52.434376 25257 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651852476208-21193-0/minicluster-data/master-0/data/instance:
uuid: "ebba074c6cca4697a4134f751768fb64"
format_stamp: "Formatted at 2026-09-29 03:17:52 on dist-test-slave-c9qh"
server_key: "5428e92855de42559e5475b6a1d1d7e7"
server_key_iv: "2666f9d4e06d2ad219366b8dd442c3e0"
server_key_version: "encryptionkey@0"
I20260929 03:17:52.434736 25257 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651852476208-21193-0/minicluster-data/master-0/wal/instance:
uuid: "ebba074c6cca4697a4134f751768fb64"
format_stamp: "Formatted at 2026-09-29 03:17:52 on dist-test-slave-c9qh"
server_key: "5428e92855de42559e5475b6a1d1d7e7"
server_key_iv: "2666f9d4e06d2ad219366b8dd442c3e0"
server_key_version: "encryptionkey@0"
I20260929 03:17:52.436071 25257 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 03:17:52.436842 25271 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:17:52.437019 25257 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 03:17:52.437129 25257 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651852476208-21193-0/minicluster-data/master-0/data,/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651852476208-21193-0/minicluster-data/master-0/wal
uuid: "ebba074c6cca4697a4134f751768fb64"
format_stamp: "Formatted at 2026-09-29 03:17:52 on dist-test-slave-c9qh"
server_key: "5428e92855de42559e5475b6a1d1d7e7"
server_key_iv: "2666f9d4e06d2ad219366b8dd442c3e0"
server_key_version: "encryptionkey@0"
I20260929 03:17:52.437208 25257 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651852476208-21193-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651852476208-21193-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651852476208-21193-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:17:52.453150 25257 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:17:52.455482 25257 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:17:52.455643 25257 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:17:52.459559 25257 rpc_server.cc:307] RPC server started. Bound to: 127.20.178.126:44497
I20260929 03:17:52.459586 25323 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.178.126:44497 every 8 connection(s)
I20260929 03:17:52.459865 25257 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651852476208-21193-0/minicluster-data/master-0/data/info.pb
I20260929 03:17:52.460774 25324 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:17:52.462981 25324 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ebba074c6cca4697a4134f751768fb64: Bootstrap starting.
I20260929 03:17:52.463593 25324 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ebba074c6cca4697a4134f751768fb64: Neither blocks nor log segments found. Creating new log.
I20260929 03:17:52.463901 25324 log.cc:824] T 00000000000000000000000000000000 P ebba074c6cca4697a4134f751768fb64: Log is configured to *not* fsync() on all Append() calls
I20260929 03:17:52.464560 25324 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ebba074c6cca4697a4134f751768fb64: No bootstrap required, opened a new log
I20260929 03:17:52.465636 21193 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuoaHbX/build/release/bin/kudu as pid 25257
I20260929 03:17:52.465720 21193 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651852476208-21193-0/minicluster-data/master-0/wal/instance
I20260929 03:17:52.465916 21193 external_mini_cluster.cc:1428] Setting key 7e02c3027ff4687fb47e5f9c8bfbfdcd
I20260929 03:17:52.466490 25324 raft_consensus.cc:348] T 00000000000000000000000000000000 P ebba074c6cca4697a4134f751768fb64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebba074c6cca4697a4134f751768fb64" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 44497 } }
I20260929 03:17:52.466682 25324 raft_consensus.cc:374] T 00000000000000000000000000000000 P ebba074c6cca4697a4134f751768fb64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:17:52.466720 25324 raft_consensus.cc:729] T 00000000000000000000000000000000 P ebba074c6cca4697a4134f751768fb64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebba074c6cca4697a4134f751768fb64, State: Initialized, Role: FOLLOWER
I20260929 03:17:52.466856 25324 consensus_queue.cc:260] T 00000000000000000000000000000000 P ebba074c6cca4697a4134f751768fb64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebba074c6cca4697a4134f751768fb64" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 44497 } }
I20260929 03:17:52.466936 25324 raft_consensus.cc:388] T 00000000000000000000000000000000 P ebba074c6cca4697a4134f751768fb64 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:17:52.466969 25324 raft_consensus.cc:482] T 00000000000000000000000000000000 P ebba074c6cca4697a4134f751768fb64 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:17:52.467021 25324 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ebba074c6cca4697a4134f751768fb64 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:17:52.467689 25324 raft_consensus.cc:504] T 00000000000000000000000000000000 P ebba074c6cca4697a4134f751768fb64 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebba074c6cca4697a4134f751768fb64" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 44497 } }
I20260929 03:17:52.467782 25324 leader_election.cc:302] T 00000000000000000000000000000000 P ebba074c6cca4697a4134f751768fb64 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ebba074c6cca4697a4134f751768fb64; no voters:
I20260929 03:17:52.467962 25324 leader_election.cc:288] T 00000000000000000000000000000000 P ebba074c6cca4697a4134f751768fb64 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:17:52.468003 25329 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ebba074c6cca4697a4134f751768fb64 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:17:52.468148 25329 raft_consensus.cc:686] T 00000000000000000000000000000000 P ebba074c6cca4697a4134f751768fb64 [term 1 LEADER]: Becoming Leader. State: Replica: ebba074c6cca4697a4134f751768fb64, State: Running, Role: LEADER
I20260929 03:17:52.468218 25324 sys_catalog.cc:564] T 00000000000000000000000000000000 P ebba074c6cca4697a4134f751768fb64 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:17:52.468274 25329 consensus_queue.cc:237] T 00000000000000000000000000000000 P ebba074c6cca4697a4134f751768fb64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebba074c6cca4697a4134f751768fb64" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 44497 } }
I20260929 03:17:52.468597 25329 sys_catalog.cc:455] T 00000000000000000000000000000000 P ebba074c6cca4697a4134f751768fb64 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ebba074c6cca4697a4134f751768fb64. Latest consensus state: current_term: 1 leader_uuid: "ebba074c6cca4697a4134f751768fb64" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebba074c6cca4697a4134f751768fb64" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 44497 } } }
I20260929 03:17:52.468606 25331 sys_catalog.cc:455] T 00000000000000000000000000000000 P ebba074c6cca4697a4134f751768fb64 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ebba074c6cca4697a4134f751768fb64" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebba074c6cca4697a4134f751768fb64" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 44497 } } }
I20260929 03:17:52.468801 25329 sys_catalog.cc:458] T 00000000000000000000000000000000 P ebba074c6cca4697a4134f751768fb64 [sys.catalog]: This master's current role is: LEADER
I20260929 03:17:52.468811 25331 sys_catalog.cc:458] T 00000000000000000000000000000000 P ebba074c6cca4697a4134f751768fb64 [sys.catalog]: This master's current role is: LEADER
I20260929 03:17:52.469096 25338 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:17:52.469792 25338 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:17:52.471377 25338 catalog_manager.cc:1347] Generated new cluster ID: 17316e8306b84ec79db1d36609afc4fe
I20260929 03:17:52.471472 25338 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 29 03:17:52 dist-test-slave-c9qh krb5kdc[25244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651872, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.178.126@KRBTEST.COM
I20260929 03:17:52.477519 25338 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:17:52.478250 25338 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:17:52.478477 25328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:52.466486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55322 (local address 127.20.178.126:44497)
0929 03:17:52.466676 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:52.466685 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:52.466718 (+ 33us) server_negotiation.cc:407] Connection header received
0929 03:17:52.467124 (+ 406us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:52.467132 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:52.467191 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:52.467339 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:17:52.467978 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:52.469554 (+ 1576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:52.470664 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:52.471258 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:52.471367 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:52.474105 (+ 2738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:17:52.474109 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:17:52.474109 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:17:52.474121 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:17:52.476117 (+ 1996us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:52.476517 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:52.476520 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:52.476520 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:52.476628 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:52.476885 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:52.476886 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:52.476887 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:52.477315 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:17:52.477977 (+ 662us) server_negotiation.cc:1035] Waiting for connection context
0929 03:17:52.478063 (+ 86us) server_negotiation.cc:299] Negotiation successful
0929 03:17:52.478238 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":42,"server-negotiator.queue_time_us":143,"thread_start_us":82,"threads_started":1}
I20260929 03:17:52.484328 25338 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ebba074c6cca4697a4134f751768fb64: Generated new TSK 0
I20260929 03:17:52.484504 25338 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.178.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.178.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.178.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.65.keytab.
Entry for principal kudu/127.20.178.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.65.keytab.
I20260929 03:17:52.517438 21193 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuoaHbX/build/release/bin/kudu
/tmp/dist-test-taskuoaHbX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651852476208-21193-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651852476208-21193-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651852476208-21193-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651852476208-21193-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.178.65:0
--local_ip_for_outbound_sockets=127.20.178.65
--webserver_interface=127.20.178.65
--webserver_port=0
--tserver_master_addrs=127.20.178.126:44497
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.65.keytab
--principal=kudu/127.20.178.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.178.84:44819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651852476208-21193-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651852476208-21193-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651852476208-21193-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:17:52.607316 25350 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:17:52.607546 25350 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:17:52.607589 25350 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:17:52.608780 25350 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:17:52.608846 25350 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:17:52.608893 25350 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.178.65
I20260929 03:17:52.610397 25350 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.178.84:44819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651852476208-21193-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651852476208-21193-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.65.keytab
--principal=kudu/127.20.178.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.178.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651852476208-21193-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.178.65
--webserver_port=0
--tserver_master_addrs=127.20.178.126:44497
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25350
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.178.65
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651852476208-21193-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:17:52.610656 25350 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:17:52.610896 25350 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:17:52.615918 25355 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:17:52.615967 25356 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:17:52.615926 25358 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:17:52.616197 25350 server_base.cc:1034] running on GCE node
I20260929 03:17:52.616492 25350 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:17:52.616760 25350 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:17:52.617919 25350 hybrid_clock.cc:648] HybridClock initialized: now 1790651872617899 us; error 36 us; skew 500 ppm
Sep 29 03:17:52 dist-test-slave-c9qh krb5kdc[25244](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651872, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:17:52.620548 25350 init.cc:377] Logged in from keytab as kudu/127.20.178.65@KRBTEST.COM (short username kudu)
I20260929 03:17:52.620944 25350 fs_manager.cc:359] Metadata directory not provided
I20260929 03:17:52.620994 25350 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:17:52.621081 25350 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:17:52.622146 25350 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651852476208-21193-0/minicluster-data/ts-0/data/instance:
uuid: "1d36eeb42cff43788b9580f64e5df922"
format_stamp: "Formatted at 2026-09-29 03:17:52 on dist-test-slave-c9qh"
server_key: "1507e61aae50d583d8a15e27f87792c8"
server_key_iv: "857d3ddb7bed0baa158b7a3be8c0f558"
server_key_version: "encryptionkey@0"
I20260929 03:17:52.622464 25350 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651852476208-21193-0/minicluster-data/ts-0/wal/instance:
uuid: "1d36eeb42cff43788b9580f64e5df922"
format_stamp: "Formatted at 2026-09-29 03:17:52 on dist-test-slave-c9qh"
server_key: "1507e61aae50d583d8a15e27f87792c8"
server_key_iv: "857d3ddb7bed0baa158b7a3be8c0f558"
server_key_version: "encryptionkey@0"
I20260929 03:17:52.623646 25350 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 03:17:52.624312 25364 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:17:52.624467 25350 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 03:17:52.624548 25350 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651852476208-21193-0/minicluster-data/ts-0/data,/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651852476208-21193-0/minicluster-data/ts-0/wal
uuid: "1d36eeb42cff43788b9580f64e5df922"
format_stamp: "Formatted at 2026-09-29 03:17:52 on dist-test-slave-c9qh"
server_key: "1507e61aae50d583d8a15e27f87792c8"
server_key_iv: "857d3ddb7bed0baa158b7a3be8c0f558"
server_key_version: "encryptionkey@0"
I20260929 03:17:52.624688 25350 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651852476208-21193-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651852476208-21193-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651852476208-21193-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:17:52.642225 25350 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:17:52.644414 25350 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:17:52.644519 25350 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:17:52.644757 25350 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:17:52.645016 25350 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:17:52.645045 25350 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:17:52.645073 25350 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:17:52.645090 25350 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:17:52.650700 25350 rpc_server.cc:307] RPC server started. Bound to: 127.20.178.65:42493
I20260929 03:17:52.650789 25477 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.178.65:42493 every 8 connection(s)
I20260929 03:17:52.651042 25350 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651852476208-21193-0/minicluster-data/ts-0/data/info.pb
I20260929 03:17:52.652426 21193 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuoaHbX/build/release/bin/kudu as pid 25350
I20260929 03:17:52.652542 21193 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651852476208-21193-0/minicluster-data/ts-0/wal/instance
I20260929 03:17:52.652771 21193 external_mini_cluster.cc:1428] Setting key 3f2dcc30847affa9f28b740dd25db8e2
Sep 29 03:17:52 dist-test-slave-c9qh krb5kdc[25244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651872, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.65@KRBTEST.COM for kudu/127.20.178.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.20.178.66@KRBTEST.COM; defaulting to no policy
I20260929 03:17:52.661142 25328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:52.651792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.65:54155 (local address 127.20.178.126:44497)
0929 03:17:52.651865 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:52.651867 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:52.652025 (+ 158us) server_negotiation.cc:407] Connection header received
0929 03:17:52.652729 (+ 704us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:52.652730 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:52.652744 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:52.652799 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:17:52.654004 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:52.654473 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:52.655584 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:52.655850 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:52.655904 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:52.658317 (+ 2413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:17:52.658318 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:17:52.658319 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:17:52.658326 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:17:52.659728 (+ 1402us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:52.660137 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:52.660139 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:52.660139 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:52.660200 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:52.660444 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:52.660445 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:52.660446 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:52.660586 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:17:52.660684 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0929 03:17:52.660962 (+ 278us) server_negotiation.cc:299] Negotiation successful
0929 03:17:52.661042 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260929 03:17:52.661314 25480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:52.651745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:44497 (local address 127.20.178.65:54155)
0929 03:17:52.651906 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0929 03:17:52.651945 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 03:17:52.652581 (+ 636us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:17:52.652875 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:17:52.652879 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:17:52.653210 (+ 331us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:17:52.653916 (+ 706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:52.653923 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:52.654564 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:52.654566 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:52.655497 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:52.655499 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:52.655934 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:52.655935 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:52.656078 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:52.656577 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:17:52.656587 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:17:52.658222 (+ 1635us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:17:52.659828 (+ 1606us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:17:52.659836 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:17:52.659838 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:17:52.660060 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:17:52.660268 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:17:52.660268 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:17:52.660269 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:17:52.660376 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:17:52.660684 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:17:52.660692 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:17:52.660828 (+ 136us) client_negotiation.cc:769] Sending connection context
0929 03:17:52.660930 (+ 102us) client_negotiation.cc:240] Negotiation successful
0929 03:17:52.661082 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":91,"threads_started":1}
I20260929 03:17:52.661823 25478 heartbeater.cc:347] Connected to a master server at 127.20.178.126:44497
I20260929 03:17:52.661921 25478 heartbeater.cc:464] Registering TS with master...
I20260929 03:17:52.662029 25478 heartbeater.cc:511] Master 127.20.178.126:44497 requested a full tablet report, sending...
I20260929 03:17:52.662624 25288 ts_manager.cc:194] Registered new tserver with Master: 1d36eeb42cff43788b9580f64e5df922 (127.20.178.65:42493)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.178.66@KRBTEST.COM" created.
I20260929 03:17:52.663360 25288 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.178.65@KRBTEST.COM'} at 127.20.178.65:54155
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.178.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.66.keytab.
Entry for principal kudu/127.20.178.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.66.keytab.
I20260929 03:17:52.677736 21193 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuoaHbX/build/release/bin/kudu
/tmp/dist-test-taskuoaHbX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651852476208-21193-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651852476208-21193-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651852476208-21193-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651852476208-21193-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.178.66:0
--local_ip_for_outbound_sockets=127.20.178.66
--webserver_interface=127.20.178.66
--webserver_port=0
--tserver_master_addrs=127.20.178.126:44497
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.66.keytab
--principal=kudu/127.20.178.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.178.84:44819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651852476208-21193-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651852476208-21193-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651852476208-21193-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:17:52.757705 25483 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:17:52.757910 25483 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:17:52.757933 25483 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:17:52.759090 25483 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:17:52.759131 25483 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:17:52.759186 25483 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.178.66
I20260929 03:17:52.760681 25483 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.178.84:44819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651852476208-21193-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651852476208-21193-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.66.keytab
--principal=kudu/127.20.178.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.178.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651852476208-21193-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.178.66
--webserver_port=0
--tserver_master_addrs=127.20.178.126:44497
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25483
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.178.66
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651852476208-21193-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:17:52.760938 25483 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:17:52.761183 25483 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:17:52.765996 25488 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:17:52.765990 25489 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:17:52.765992 25491 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:17:52.766098 25483 server_base.cc:1034] running on GCE node
I20260929 03:17:52.766374 25483 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:17:52.766618 25483 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:17:52.767777 25483 hybrid_clock.cc:648] HybridClock initialized: now 1790651872767758 us; error 36 us; skew 500 ppm
Sep 29 03:17:52 dist-test-slave-c9qh krb5kdc[25244](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651872, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:17:52.770478 25483 init.cc:377] Logged in from keytab as kudu/127.20.178.66@KRBTEST.COM (short username kudu)
I20260929 03:17:52.770848 25483 fs_manager.cc:359] Metadata directory not provided
I20260929 03:17:52.770919 25483 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:17:52.771059 25483 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:17:52.772076 25483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651852476208-21193-0/minicluster-data/ts-1/data/instance:
uuid: "7ce8504013b94949a1a0a78b23384259"
format_stamp: "Formatted at 2026-09-29 03:17:52 on dist-test-slave-c9qh"
server_key: "2b4c996c7e7a1a4904b01c56cefe87d9"
server_key_iv: "4b621654525332d2cfb834fcda352fde"
server_key_version: "encryptionkey@0"
I20260929 03:17:52.772526 25483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651852476208-21193-0/minicluster-data/ts-1/wal/instance:
uuid: "7ce8504013b94949a1a0a78b23384259"
format_stamp: "Formatted at 2026-09-29 03:17:52 on dist-test-slave-c9qh"
server_key: "2b4c996c7e7a1a4904b01c56cefe87d9"
server_key_iv: "4b621654525332d2cfb834fcda352fde"
server_key_version: "encryptionkey@0"
I20260929 03:17:52.773818 25483 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260929 03:17:52.774519 25497 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:17:52.774686 25483 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 03:17:52.774785 25483 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651852476208-21193-0/minicluster-data/ts-1/data,/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651852476208-21193-0/minicluster-data/ts-1/wal
uuid: "7ce8504013b94949a1a0a78b23384259"
format_stamp: "Formatted at 2026-09-29 03:17:52 on dist-test-slave-c9qh"
server_key: "2b4c996c7e7a1a4904b01c56cefe87d9"
server_key_iv: "4b621654525332d2cfb834fcda352fde"
server_key_version: "encryptionkey@0"
I20260929 03:17:52.774873 25483 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651852476208-21193-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651852476208-21193-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651852476208-21193-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:17:52.791019 25483 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:17:52.793077 25483 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:17:52.793247 25483 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:17:52.793486 25483 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:17:52.793812 25483 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:17:52.793869 25483 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:17:52.793905 25483 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:17:52.793932 25483 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:17:52.799719 25483 rpc_server.cc:307] RPC server started. Bound to: 127.20.178.66:43787
I20260929 03:17:52.799773 25610 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.178.66:43787 every 8 connection(s)
I20260929 03:17:52.800025 25483 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651852476208-21193-0/minicluster-data/ts-1/data/info.pb
I20260929 03:17:52.802027 21193 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuoaHbX/build/release/bin/kudu as pid 25483
I20260929 03:17:52.802155 21193 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651852476208-21193-0/minicluster-data/ts-1/wal/instance
I20260929 03:17:52.802331 21193 external_mini_cluster.cc:1428] Setting key 0166b346545030632e9a367ce4d4adf3
Sep 29 03:17:52 dist-test-slave-c9qh krb5kdc[25244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651872, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.66@KRBTEST.COM for kudu/127.20.178.126@KRBTEST.COM
I20260929 03:17:52.810348 25328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:52.800750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.66:50789 (local address 127.20.178.126:44497)
0929 03:17:52.800798 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:52.800800 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:52.801050 (+ 250us) server_negotiation.cc:407] Connection header received
0929 03:17:52.801499 (+ 449us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:52.801500 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:52.801510 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:52.801565 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:17:52.802641 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:52.803377 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:52.804565 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:52.804769 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:52.804826 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:52.807366 (+ 2540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:17:52.807367 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:17:52.807368 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:17:52.807374 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:17:52.808872 (+ 1498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:52.809354 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:52.809355 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:52.809355 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:52.809404 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:52.809696 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:52.809697 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:52.809697 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:52.809800 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:17:52.809850 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0929 03:17:52.810200 (+ 350us) server_negotiation.cc:299] Negotiation successful
0929 03:17:52.810267 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260929 03:17:52.810652 25613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:52.800781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:44497 (local address 127.20.178.66:50789)
0929 03:17:52.800927 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0929 03:17:52.800962 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0929 03:17:52.801409 (+ 447us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:17:52.801633 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:17:52.801637 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:17:52.801882 (+ 245us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:17:52.802510 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:52.802518 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:52.803472 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:52.803473 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:52.804461 (+ 988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:52.804475 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:52.804850 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:52.804851 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:52.805001 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:52.805504 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:17:52.805514 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:17:52.807275 (+ 1761us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:17:52.808953 (+ 1678us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:17:52.808961 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:17:52.808963 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:17:52.809277 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:17:52.809470 (+ 193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:17:52.809472 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:17:52.809472 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:17:52.809623 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:17:52.809866 (+ 243us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:17:52.809872 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:17:52.810043 (+ 171us) client_negotiation.cc:769] Sending connection context
0929 03:17:52.810185 (+ 142us) client_negotiation.cc:240] Negotiation successful
0929 03:17:52.810369 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":89,"threads_started":1}
WARNING: no policy specified for kudu/127.20.178.67@KRBTEST.COM; defaulting to no policy
I20260929 03:17:52.811240 25611 heartbeater.cc:347] Connected to a master server at 127.20.178.126:44497
I20260929 03:17:52.811367 25611 heartbeater.cc:464] Registering TS with master...
I20260929 03:17:52.811512 25611 heartbeater.cc:511] Master 127.20.178.126:44497 requested a full tablet report, sending...
I20260929 03:17:52.811975 25288 ts_manager.cc:194] Registered new tserver with Master: 7ce8504013b94949a1a0a78b23384259 (127.20.178.66:43787)
I20260929 03:17:52.812461 25288 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.178.66@KRBTEST.COM'} at 127.20.178.66:50789
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.178.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.178.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.67.keytab.
Entry for principal kudu/127.20.178.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.67.keytab.
I20260929 03:17:52.827785 21193 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuoaHbX/build/release/bin/kudu
/tmp/dist-test-taskuoaHbX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651852476208-21193-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651852476208-21193-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651852476208-21193-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651852476208-21193-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.178.67:0
--local_ip_for_outbound_sockets=127.20.178.67
--webserver_interface=127.20.178.67
--webserver_port=0
--tserver_master_addrs=127.20.178.126:44497
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.67.keytab
--principal=kudu/127.20.178.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.178.84:44819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651852476208-21193-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651852476208-21193-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651852476208-21193-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:17:52.909376 25616 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:17:52.909569 25616 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:17:52.909591 25616 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:17:52.910667 25616 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:17:52.910703 25616 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:17:52.910729 25616 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.178.67
I20260929 03:17:52.912220 25616 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.178.84:44819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651852476208-21193-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651852476208-21193-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.67.keytab
--principal=kudu/127.20.178.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.178.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651852476208-21193-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.178.67
--webserver_port=0
--tserver_master_addrs=127.20.178.126:44497
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25616
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.178.67
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651852476208-21193-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:17:52.912406 25616 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:17:52.912591 25616 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:17:52.917649 25622 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:17:52.917667 25621 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:17:52.917722 25616 server_base.cc:1034] running on GCE node
W20260929 03:17:52.917649 25624 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:17:52.918073 25616 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:17:52.918272 25616 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:17:52.919417 25616 hybrid_clock.cc:648] HybridClock initialized: now 1790651872919404 us; error 27 us; skew 500 ppm
Sep 29 03:17:52 dist-test-slave-c9qh krb5kdc[25244](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651872, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:17:52.922183 25616 init.cc:377] Logged in from keytab as kudu/127.20.178.67@KRBTEST.COM (short username kudu)
I20260929 03:17:52.922525 25616 fs_manager.cc:359] Metadata directory not provided
I20260929 03:17:52.922573 25616 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:17:52.922700 25616 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:17:52.923753 25616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651852476208-21193-0/minicluster-data/ts-2/data/instance:
uuid: "2f543eefd16843778e940651a1560ae2"
format_stamp: "Formatted at 2026-09-29 03:17:52 on dist-test-slave-c9qh"
server_key: "e57c9d359f5e4dacbd40de53617da086"
server_key_iv: "71ee40ab3134e2d8edb2045c5dc86a7f"
server_key_version: "encryptionkey@0"
I20260929 03:17:52.924108 25616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651852476208-21193-0/minicluster-data/ts-2/wal/instance:
uuid: "2f543eefd16843778e940651a1560ae2"
format_stamp: "Formatted at 2026-09-29 03:17:52 on dist-test-slave-c9qh"
server_key: "e57c9d359f5e4dacbd40de53617da086"
server_key_iv: "71ee40ab3134e2d8edb2045c5dc86a7f"
server_key_version: "encryptionkey@0"
I20260929 03:17:52.925483 25616 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 03:17:52.926261 25630 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:17:52.926476 25616 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260929 03:17:52.926548 25616 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651852476208-21193-0/minicluster-data/ts-2/data,/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651852476208-21193-0/minicluster-data/ts-2/wal
uuid: "2f543eefd16843778e940651a1560ae2"
format_stamp: "Formatted at 2026-09-29 03:17:52 on dist-test-slave-c9qh"
server_key: "e57c9d359f5e4dacbd40de53617da086"
server_key_iv: "71ee40ab3134e2d8edb2045c5dc86a7f"
server_key_version: "encryptionkey@0"
I20260929 03:17:52.926627 25616 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651852476208-21193-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651852476208-21193-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651852476208-21193-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:17:52.939620 25616 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:17:52.941578 25616 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:17:52.941730 25616 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:17:52.941936 25616 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:17:52.942230 25616 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:17:52.942289 25616 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:17:52.942332 25616 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:17:52.942358 25616 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:17:52.948079 25616 rpc_server.cc:307] RPC server started. Bound to: 127.20.178.67:37945
I20260929 03:17:52.948143 25743 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.178.67:37945 every 8 connection(s)
I20260929 03:17:52.948417 25616 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651852476208-21193-0/minicluster-data/ts-2/data/info.pb
I20260929 03:17:52.952556 21193 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuoaHbX/build/release/bin/kudu as pid 25616
I20260929 03:17:52.952709 21193 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651852476208-21193-0/minicluster-data/ts-2/wal/instance
I20260929 03:17:52.952898 21193 external_mini_cluster.cc:1428] Setting key cf56b71fb5746786976af4794b578aac
Sep 29 03:17:52 dist-test-slave-c9qh krb5kdc[25244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651872, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.67@KRBTEST.COM for kudu/127.20.178.126@KRBTEST.COM
I20260929 03:17:52.957144 25328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:52.949135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.67:42367 (local address 127.20.178.126:44497)
0929 03:17:52.949213 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:52.949215 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:52.949370 (+ 155us) server_negotiation.cc:407] Connection header received
0929 03:17:52.949813 (+ 443us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:52.949814 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:52.949825 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:52.949877 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:17:52.950726 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:52.951207 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:52.952127 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:52.952305 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:52.952350 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:52.954661 (+ 2311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:17:52.954662 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:17:52.954662 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:17:52.954668 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:17:52.956017 (+ 1349us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:52.956354 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:52.956355 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:52.956356 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:52.956403 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:52.956608 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:52.956609 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:52.956609 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:52.956745 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:17:52.956808 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0929 03:17:52.957025 (+ 217us) server_negotiation.cc:299] Negotiation successful
0929 03:17:52.957082 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 03:17:52.957366 25746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:52.949095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:44497 (local address 127.20.178.67:42367)
0929 03:17:52.949256 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0929 03:17:52.949290 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0929 03:17:52.949720 (+ 430us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:17:52.949928 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:17:52.949931 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:17:52.950189 (+ 258us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:17:52.950653 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:52.950658 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:52.951281 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:52.951281 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:52.952048 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:52.952050 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:52.952414 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:52.952415 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:52.952575 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:52.953094 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:17:52.953126 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:17:52.954587 (+ 1461us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:17:52.956087 (+ 1500us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:17:52.956092 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:17:52.956093 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:17:52.956290 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:17:52.956459 (+ 169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:17:52.956460 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:17:52.956460 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:17:52.956547 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:17:52.956810 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:17:52.956813 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:17:52.956916 (+ 103us) client_negotiation.cc:769] Sending connection context
0929 03:17:52.956997 (+ 81us) client_negotiation.cc:240] Negotiation successful
0929 03:17:52.957120 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":93,"threads_started":1}
I20260929 03:17:52.957865 25744 heartbeater.cc:347] Connected to a master server at 127.20.178.126:44497
I20260929 03:17:52.957937 25744 heartbeater.cc:464] Registering TS with master...
I20260929 03:17:52.958096 25744 heartbeater.cc:511] Master 127.20.178.126:44497 requested a full tablet report, sending...
I20260929 03:17:52.958525 25288 ts_manager.cc:194] Registered new tserver with Master: 2f543eefd16843778e940651a1560ae2 (127.20.178.67:37945)
I20260929 03:17:52.958889 25288 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.178.67@KRBTEST.COM'} at 127.20.178.67:42367
I20260929 03:17:52.964701 21193 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 03:17:52.971565 25328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:52.965954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55336 (local address 127.20.178.126:44497)
0929 03:17:52.966019 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:52.966021 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:52.966099 (+ 78us) server_negotiation.cc:407] Connection header received
0929 03:17:52.966135 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:52.966136 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:52.966144 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:52.966195 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:17:52.966699 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:52.967143 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:52.967923 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:52.968110 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:52.968172 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:52.969447 (+ 1275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:17:52.969448 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:17:52.969449 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:17:52.969454 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:17:52.970660 (+ 1206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:52.970952 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:52.970953 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:52.970953 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:52.971000 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:52.971151 (+ 151us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:52.971151 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:52.971152 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:52.971271 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:17:52.971340 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0929 03:17:52.971425 (+ 85us) server_negotiation.cc:299] Negotiation successful
0929 03:17:52.971479 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260929 03:17:52.972982 25288 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55336:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 03:17:52.974483 25288 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 03:17:52.981179 25753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:52.976524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.67:37945 (local address 127.0.0.1:44730)
0929 03:17:52.977306 (+ 782us) negotiation.cc:107] Waiting for socket to connect
0929 03:17:52.977315 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 03:17:52.977361 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:17:52.977568 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:17:52.977569 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:17:52.977571 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:17:52.977623 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:52.977624 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:52.978924 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:52.978924 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:52.979928 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:52.979931 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:52.980887 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:52.980888 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:52.980963 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:52.980976 (+ 13us) client_negotiation.cc:769] Sending connection context
0929 03:17:52.981042 (+ 66us) client_negotiation.cc:240] Negotiation successful
0929 03:17:52.981100 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":771,"mutex_wait_us":46,"thread_start_us":49,"threads_started":1}
I20260929 03:17:52.981551 25755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:52.976736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44730 (local address 127.20.178.67:37945)
0929 03:17:52.976910 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:52.976915 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:52.977419 (+ 504us) server_negotiation.cc:407] Connection header received
0929 03:17:52.977462 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:52.977464 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:52.977477 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:52.977569 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:17:52.977693 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:52.978791 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:52.980032 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:52.980731 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:52.980797 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:52.980853 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0929 03:17:52.981321 (+ 468us) server_negotiation.cc:299] Negotiation successful
0929 03:17:52.981365 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":87,"threads_started":1}
I20260929 03:17:52.981657 25752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:52.976504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.65:42493 (local address 127.0.0.1:55942)
0929 03:17:52.977137 (+ 633us) negotiation.cc:107] Waiting for socket to connect
0929 03:17:52.977144 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0929 03:17:52.977203 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:17:52.977704 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:17:52.977705 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:17:52.977706 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:17:52.977734 (+ 28us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:52.977735 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:52.978924 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:52.978924 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:52.980240 (+ 1316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:52.980242 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:52.981438 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:52.981439 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:52.981503 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:52.981505 (+ 2us) client_negotiation.cc:769] Sending connection context
0929 03:17:52.981549 (+ 44us) client_negotiation.cc:240] Negotiation successful
0929 03:17:52.981587 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":625,"mutex_wait_us":83,"thread_start_us":54,"threads_started":1}
I20260929 03:17:52.981907 25757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:52.976761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34742 (local address 127.20.178.66:43787)
0929 03:17:52.976954 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:52.976958 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:52.977048 (+ 90us) server_negotiation.cc:407] Connection header received
0929 03:17:52.977144 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:52.977146 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:52.977159 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:52.977304 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:17:52.977622 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:52.978729 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:52.980331 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:52.981066 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:52.981143 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:52.981195 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0929 03:17:52.981792 (+ 597us) server_negotiation.cc:299] Negotiation successful
0929 03:17:52.981833 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":101,"threads_started":1}
I20260929 03:17:52.981930 25756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:52.976753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55942 (local address 127.20.178.65:42493)
0929 03:17:52.977010 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:52.977012 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:52.977206 (+ 194us) server_negotiation.cc:407] Connection header received
0929 03:17:52.977285 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:52.977288 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:52.977304 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:52.977361 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:17:52.977806 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:52.978838 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:52.980709 (+ 1871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:52.981237 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:52.981338 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:52.981403 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0929 03:17:52.981771 (+ 368us) server_negotiation.cc:299] Negotiation successful
0929 03:17:52.981833 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":234,"thread_start_us":76,"threads_started":1}
I20260929 03:17:52.982034 25754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:52.976504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.66:43787 (local address 127.0.0.1:34742)
0929 03:17:52.976955 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0929 03:17:52.977003 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0929 03:17:52.977065 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:17:52.977462 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:17:52.977465 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:17:52.977468 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:17:52.977526 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:52.977528 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:52.978829 (+ 1301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:52.978830 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:52.980239 (+ 1409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:52.980242 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:52.981150 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:52.981151 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:52.981433 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:52.981439 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 03:17:52.981483 (+ 44us) client_negotiation.cc:240] Negotiation successful
0929 03:17:52.981529 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":429,"mutex_wait_us":66,"thread_start_us":52,"threads_started":1}
I20260929 03:17:52.982251 25678 tablet_service.cc:1467] Processing CreateTablet for tablet e8abb85a646a4ec7b62926041ee8d894 (DEFAULT_TABLE table=test-table [id=46a2e9231be7476f87e31e07f99d266a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:17:52.982491 25412 tablet_service.cc:1467] Processing CreateTablet for tablet e8abb85a646a4ec7b62926041ee8d894 (DEFAULT_TABLE table=test-table [id=46a2e9231be7476f87e31e07f99d266a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:17:52.982686 25678 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8abb85a646a4ec7b62926041ee8d894. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:17:52.982769 25412 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8abb85a646a4ec7b62926041ee8d894. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:17:52.984946 25545 tablet_service.cc:1467] Processing CreateTablet for tablet e8abb85a646a4ec7b62926041ee8d894 (DEFAULT_TABLE table=test-table [id=46a2e9231be7476f87e31e07f99d266a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:17:52.985292 25545 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8abb85a646a4ec7b62926041ee8d894. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:17:52.985767 25758 tablet_bootstrap.cc:492] T e8abb85a646a4ec7b62926041ee8d894 P 1d36eeb42cff43788b9580f64e5df922: Bootstrap starting.
I20260929 03:17:52.986413 25758 tablet_bootstrap.cc:654] T e8abb85a646a4ec7b62926041ee8d894 P 1d36eeb42cff43788b9580f64e5df922: Neither blocks nor log segments found. Creating new log.
I20260929 03:17:52.987041 25758 log.cc:824] T e8abb85a646a4ec7b62926041ee8d894 P 1d36eeb42cff43788b9580f64e5df922: Log is configured to *not* fsync() on all Append() calls
I20260929 03:17:52.987406 25759 tablet_bootstrap.cc:492] T e8abb85a646a4ec7b62926041ee8d894 P 2f543eefd16843778e940651a1560ae2: Bootstrap starting.
I20260929 03:17:52.987986 25758 tablet_bootstrap.cc:492] T e8abb85a646a4ec7b62926041ee8d894 P 1d36eeb42cff43788b9580f64e5df922: No bootstrap required, opened a new log
I20260929 03:17:52.988072 25759 tablet_bootstrap.cc:654] T e8abb85a646a4ec7b62926041ee8d894 P 2f543eefd16843778e940651a1560ae2: Neither blocks nor log segments found. Creating new log.
I20260929 03:17:52.988090 25758 ts_tablet_manager.cc:1397] T e8abb85a646a4ec7b62926041ee8d894 P 1d36eeb42cff43788b9580f64e5df922: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 03:17:52.988675 25759 log.cc:824] T e8abb85a646a4ec7b62926041ee8d894 P 2f543eefd16843778e940651a1560ae2: Log is configured to *not* fsync() on all Append() calls
I20260929 03:17:52.988958 25761 tablet_bootstrap.cc:492] T e8abb85a646a4ec7b62926041ee8d894 P 7ce8504013b94949a1a0a78b23384259: Bootstrap starting.
I20260929 03:17:52.989480 25761 tablet_bootstrap.cc:654] T e8abb85a646a4ec7b62926041ee8d894 P 7ce8504013b94949a1a0a78b23384259: Neither blocks nor log segments found. Creating new log.
I20260929 03:17:52.989756 25758 raft_consensus.cc:348] T e8abb85a646a4ec7b62926041ee8d894 P 1d36eeb42cff43788b9580f64e5df922 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ce8504013b94949a1a0a78b23384259" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 43787 } } peers { permanent_uuid: "1d36eeb42cff43788b9580f64e5df922" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 42493 } } peers { permanent_uuid: "2f543eefd16843778e940651a1560ae2" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 37945 } }
I20260929 03:17:52.989907 25758 raft_consensus.cc:374] T e8abb85a646a4ec7b62926041ee8d894 P 1d36eeb42cff43788b9580f64e5df922 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:17:52.989951 25758 raft_consensus.cc:729] T e8abb85a646a4ec7b62926041ee8d894 P 1d36eeb42cff43788b9580f64e5df922 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d36eeb42cff43788b9580f64e5df922, State: Initialized, Role: FOLLOWER
I20260929 03:17:52.989996 25761 log.cc:824] T e8abb85a646a4ec7b62926041ee8d894 P 7ce8504013b94949a1a0a78b23384259: Log is configured to *not* fsync() on all Append() calls
I20260929 03:17:52.990099 25758 consensus_queue.cc:260] T e8abb85a646a4ec7b62926041ee8d894 P 1d36eeb42cff43788b9580f64e5df922 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ce8504013b94949a1a0a78b23384259" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 43787 } } peers { permanent_uuid: "1d36eeb42cff43788b9580f64e5df922" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 42493 } } peers { permanent_uuid: "2f543eefd16843778e940651a1560ae2" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 37945 } }
I20260929 03:17:52.990347 25758 ts_tablet_manager.cc:1428] T e8abb85a646a4ec7b62926041ee8d894 P 1d36eeb42cff43788b9580f64e5df922: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260929 03:17:52.990494 25478 heartbeater.cc:503] Master 127.20.178.126:44497 was elected leader, sending a full tablet report...
I20260929 03:17:52.990824 25759 tablet_bootstrap.cc:492] T e8abb85a646a4ec7b62926041ee8d894 P 2f543eefd16843778e940651a1560ae2: No bootstrap required, opened a new log
I20260929 03:17:52.990895 25759 ts_tablet_manager.cc:1397] T e8abb85a646a4ec7b62926041ee8d894 P 2f543eefd16843778e940651a1560ae2: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20260929 03:17:52.991080 25761 tablet_bootstrap.cc:492] T e8abb85a646a4ec7b62926041ee8d894 P 7ce8504013b94949a1a0a78b23384259: No bootstrap required, opened a new log
I20260929 03:17:52.991150 25761 ts_tablet_manager.cc:1397] T e8abb85a646a4ec7b62926041ee8d894 P 7ce8504013b94949a1a0a78b23384259: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 03:17:52.992798 25759 raft_consensus.cc:348] T e8abb85a646a4ec7b62926041ee8d894 P 2f543eefd16843778e940651a1560ae2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ce8504013b94949a1a0a78b23384259" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 43787 } } peers { permanent_uuid: "1d36eeb42cff43788b9580f64e5df922" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 42493 } } peers { permanent_uuid: "2f543eefd16843778e940651a1560ae2" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 37945 } }
I20260929 03:17:52.992944 25759 raft_consensus.cc:374] T e8abb85a646a4ec7b62926041ee8d894 P 2f543eefd16843778e940651a1560ae2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:17:52.992901 25761 raft_consensus.cc:348] T e8abb85a646a4ec7b62926041ee8d894 P 7ce8504013b94949a1a0a78b23384259 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ce8504013b94949a1a0a78b23384259" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 43787 } } peers { permanent_uuid: "1d36eeb42cff43788b9580f64e5df922" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 42493 } } peers { permanent_uuid: "2f543eefd16843778e940651a1560ae2" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 37945 } }
I20260929 03:17:52.992985 25759 raft_consensus.cc:729] T e8abb85a646a4ec7b62926041ee8d894 P 2f543eefd16843778e940651a1560ae2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f543eefd16843778e940651a1560ae2, State: Initialized, Role: FOLLOWER
I20260929 03:17:52.993031 25761 raft_consensus.cc:374] T e8abb85a646a4ec7b62926041ee8d894 P 7ce8504013b94949a1a0a78b23384259 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:17:52.993058 25761 raft_consensus.cc:729] T e8abb85a646a4ec7b62926041ee8d894 P 7ce8504013b94949a1a0a78b23384259 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ce8504013b94949a1a0a78b23384259, State: Initialized, Role: FOLLOWER
I20260929 03:17:52.993110 25759 consensus_queue.cc:260] T e8abb85a646a4ec7b62926041ee8d894 P 2f543eefd16843778e940651a1560ae2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ce8504013b94949a1a0a78b23384259" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 43787 } } peers { permanent_uuid: "1d36eeb42cff43788b9580f64e5df922" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 42493 } } peers { permanent_uuid: "2f543eefd16843778e940651a1560ae2" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 37945 } }
I20260929 03:17:52.993185 25761 consensus_queue.cc:260] T e8abb85a646a4ec7b62926041ee8d894 P 7ce8504013b94949a1a0a78b23384259 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ce8504013b94949a1a0a78b23384259" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 43787 } } peers { permanent_uuid: "1d36eeb42cff43788b9580f64e5df922" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 42493 } } peers { permanent_uuid: "2f543eefd16843778e940651a1560ae2" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 37945 } }
I20260929 03:17:52.993413 25759 ts_tablet_manager.cc:1428] T e8abb85a646a4ec7b62926041ee8d894 P 2f543eefd16843778e940651a1560ae2: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260929 03:17:52.993425 25744 heartbeater.cc:503] Master 127.20.178.126:44497 was elected leader, sending a full tablet report...
I20260929 03:17:52.993468 25611 heartbeater.cc:503] Master 127.20.178.126:44497 was elected leader, sending a full tablet report...
I20260929 03:17:52.993475 25761 ts_tablet_manager.cc:1428] T e8abb85a646a4ec7b62926041ee8d894 P 7ce8504013b94949a1a0a78b23384259: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260929 03:17:53.014518 25766 raft_consensus.cc:482] T e8abb85a646a4ec7b62926041ee8d894 P 7ce8504013b94949a1a0a78b23384259 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:17:53.014634 25766 raft_consensus.cc:504] T e8abb85a646a4ec7b62926041ee8d894 P 7ce8504013b94949a1a0a78b23384259 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ce8504013b94949a1a0a78b23384259" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 43787 } } peers { permanent_uuid: "1d36eeb42cff43788b9580f64e5df922" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 42493 } } peers { permanent_uuid: "2f543eefd16843778e940651a1560ae2" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 37945 } }
I20260929 03:17:53.015100 25766 leader_election.cc:288] T e8abb85a646a4ec7b62926041ee8d894 P 7ce8504013b94949a1a0a78b23384259 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1d36eeb42cff43788b9580f64e5df922 (127.20.178.65:42493), 2f543eefd16843778e940651a1560ae2 (127.20.178.67:37945)
I20260929 03:17:53.018074 25613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:53.015237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.67:37945 (local address 127.20.178.66:39269)
0929 03:17:53.015286 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0929 03:17:53.015297 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 03:17:53.015337 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:17:53.015516 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:17:53.015518 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:17:53.015520 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:17:53.015601 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:53.015603 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:53.016335 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:53.016336 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:53.017273 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:53.017275 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:53.017839 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:53.017840 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:53.017926 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:53.017930 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 03:17:53.017983 (+ 53us) client_negotiation.cc:240] Negotiation successful
0929 03:17:53.018012 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30,"mutex_wait_us":1}
I20260929 03:17:53.018229 25755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:53.015302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.66:39269 (local address 127.20.178.67:37945)
0929 03:17:53.015346 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:53.015348 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:53.015390 (+ 42us) server_negotiation.cc:407] Connection header received
0929 03:17:53.015424 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:53.015425 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:53.015432 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:53.015496 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:17:53.015700 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:53.016249 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:53.017366 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:53.017758 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:53.017818 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:53.017825 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0929 03:17:53.018112 (+ 287us) server_negotiation.cc:299] Negotiation successful
0929 03:17:53.018159 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260929 03:17:53.018316 25756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:53.015318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.66:58037 (local address 127.20.178.65:42493)
0929 03:17:53.015371 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:53.015373 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:53.015566 (+ 193us) server_negotiation.cc:407] Connection header received
0929 03:17:53.015603 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:53.015604 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:53.015618 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:53.015675 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:17:53.015879 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:53.016358 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:53.017406 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:53.017820 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:53.017894 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:53.017902 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0929 03:17:53.018196 (+ 294us) server_negotiation.cc:299] Negotiation successful
0929 03:17:53.018236 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260929 03:17:53.018556 25767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:53.015237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.65:42493 (local address 127.20.178.66:58037)
0929 03:17:53.015513 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0929 03:17:53.015521 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0929 03:17:53.015565 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:17:53.015692 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:17:53.015693 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:17:53.015695 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:17:53.015801 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:53.015802 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:53.016439 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:53.016440 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:53.017278 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:53.017280 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:53.017915 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:53.017916 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:53.018098 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:53.018104 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 03:17:53.018436 (+ 332us) client_negotiation.cc:240] Negotiation successful
0929 03:17:53.018486 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"mutex_wait_us":76,"thread_start_us":150,"threads_started":1}
I20260929 03:17:53.018575 25698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8abb85a646a4ec7b62926041ee8d894" candidate_uuid: "7ce8504013b94949a1a0a78b23384259" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f543eefd16843778e940651a1560ae2" is_pre_election: true
I20260929 03:17:53.018748 25698 raft_consensus.cc:2454] T e8abb85a646a4ec7b62926041ee8d894 P 2f543eefd16843778e940651a1560ae2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ce8504013b94949a1a0a78b23384259 in term 0.
I20260929 03:17:53.018967 25432 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8abb85a646a4ec7b62926041ee8d894" candidate_uuid: "7ce8504013b94949a1a0a78b23384259" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d36eeb42cff43788b9580f64e5df922" is_pre_election: true
I20260929 03:17:53.019033 25501 leader_election.cc:302] T e8abb85a646a4ec7b62926041ee8d894 P 7ce8504013b94949a1a0a78b23384259 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f543eefd16843778e940651a1560ae2, 7ce8504013b94949a1a0a78b23384259; no voters:
I20260929 03:17:53.019109 25432 raft_consensus.cc:2454] T e8abb85a646a4ec7b62926041ee8d894 P 1d36eeb42cff43788b9580f64e5df922 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ce8504013b94949a1a0a78b23384259 in term 0.
I20260929 03:17:53.019162 25766 raft_consensus.cc:2781] T e8abb85a646a4ec7b62926041ee8d894 P 7ce8504013b94949a1a0a78b23384259 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 03:17:53.019225 25766 raft_consensus.cc:482] T e8abb85a646a4ec7b62926041ee8d894 P 7ce8504013b94949a1a0a78b23384259 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 03:17:53.019256 25766 raft_consensus.cc:3037] T e8abb85a646a4ec7b62926041ee8d894 P 7ce8504013b94949a1a0a78b23384259 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:17:53.019953 25766 raft_consensus.cc:504] T e8abb85a646a4ec7b62926041ee8d894 P 7ce8504013b94949a1a0a78b23384259 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ce8504013b94949a1a0a78b23384259" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 43787 } } peers { permanent_uuid: "1d36eeb42cff43788b9580f64e5df922" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 42493 } } peers { permanent_uuid: "2f543eefd16843778e940651a1560ae2" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 37945 } }
I20260929 03:17:53.020112 25766 leader_election.cc:288] T e8abb85a646a4ec7b62926041ee8d894 P 7ce8504013b94949a1a0a78b23384259 [CANDIDATE]: Term 1 election: Requested vote from peers 1d36eeb42cff43788b9580f64e5df922 (127.20.178.65:42493), 2f543eefd16843778e940651a1560ae2 (127.20.178.67:37945)
I20260929 03:17:53.020253 25432 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8abb85a646a4ec7b62926041ee8d894" candidate_uuid: "7ce8504013b94949a1a0a78b23384259" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d36eeb42cff43788b9580f64e5df922"
I20260929 03:17:53.020294 25698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8abb85a646a4ec7b62926041ee8d894" candidate_uuid: "7ce8504013b94949a1a0a78b23384259" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f543eefd16843778e940651a1560ae2"
I20260929 03:17:53.020357 25432 raft_consensus.cc:3037] T e8abb85a646a4ec7b62926041ee8d894 P 1d36eeb42cff43788b9580f64e5df922 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:17:53.020358 25698 raft_consensus.cc:3037] T e8abb85a646a4ec7b62926041ee8d894 P 2f543eefd16843778e940651a1560ae2 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:17:53.021140 25432 raft_consensus.cc:2454] T e8abb85a646a4ec7b62926041ee8d894 P 1d36eeb42cff43788b9580f64e5df922 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ce8504013b94949a1a0a78b23384259 in term 1.
I20260929 03:17:53.021167 25698 raft_consensus.cc:2454] T e8abb85a646a4ec7b62926041ee8d894 P 2f543eefd16843778e940651a1560ae2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ce8504013b94949a1a0a78b23384259 in term 1.
I20260929 03:17:53.021335 25498 leader_election.cc:302] T e8abb85a646a4ec7b62926041ee8d894 P 7ce8504013b94949a1a0a78b23384259 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d36eeb42cff43788b9580f64e5df922, 7ce8504013b94949a1a0a78b23384259; no voters:
I20260929 03:17:53.021431 25766 raft_consensus.cc:2781] T e8abb85a646a4ec7b62926041ee8d894 P 7ce8504013b94949a1a0a78b23384259 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:17:53.021647 25766 raft_consensus.cc:686] T e8abb85a646a4ec7b62926041ee8d894 P 7ce8504013b94949a1a0a78b23384259 [term 1 LEADER]: Becoming Leader. State: Replica: 7ce8504013b94949a1a0a78b23384259, State: Running, Role: LEADER
I20260929 03:17:53.021847 25766 consensus_queue.cc:237] T e8abb85a646a4ec7b62926041ee8d894 P 7ce8504013b94949a1a0a78b23384259 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ce8504013b94949a1a0a78b23384259" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 43787 } } peers { permanent_uuid: "1d36eeb42cff43788b9580f64e5df922" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 42493 } } peers { permanent_uuid: "2f543eefd16843778e940651a1560ae2" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 37945 } }
I20260929 03:17:53.022832 25288 catalog_manager.cc:5495] T e8abb85a646a4ec7b62926041ee8d894 P 7ce8504013b94949a1a0a78b23384259 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7ce8504013b94949a1a0a78b23384259 (127.20.178.66). New cstate: current_term: 1 leader_uuid: "7ce8504013b94949a1a0a78b23384259" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ce8504013b94949a1a0a78b23384259" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 43787 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1d36eeb42cff43788b9580f64e5df922" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 42493 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f543eefd16843778e940651a1560ae2" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 37945 } health_report { overall_health: UNKNOWN } } }
I20260929 03:17:53.037942 25757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:53.035533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34758 (local address 127.20.178.66:43787)
0929 03:17:53.035592 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:53.035594 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:53.035645 (+ 51us) server_negotiation.cc:407] Connection header received
0929 03:17:53.035708 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:53.035709 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:53.035724 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:53.035787 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:17:53.035894 (+ 107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:53.036346 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:53.037191 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:53.037380 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:53.037431 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:53.037620 (+ 189us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:17:53.037713 (+ 93us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:17:53.037773 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0929 03:17:53.037814 (+ 41us) server_negotiation.cc:299] Negotiation successful
0929 03:17:53.037830 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260929 03:17:53.039603 25698 raft_consensus.cc:1261] T e8abb85a646a4ec7b62926041ee8d894 P 2f543eefd16843778e940651a1560ae2 [term 1 FOLLOWER]: Refusing update from remote peer 7ce8504013b94949a1a0a78b23384259: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:17:53.039608 25432 raft_consensus.cc:1261] T e8abb85a646a4ec7b62926041ee8d894 P 1d36eeb42cff43788b9580f64e5df922 [term 1 FOLLOWER]: Refusing update from remote peer 7ce8504013b94949a1a0a78b23384259: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:17:53.039969 25766 consensus_queue.cc:1035] T e8abb85a646a4ec7b62926041ee8d894 P 7ce8504013b94949a1a0a78b23384259 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d36eeb42cff43788b9580f64e5df922" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 42493 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:17:53.040124 25768 consensus_queue.cc:1035] T e8abb85a646a4ec7b62926041ee8d894 P 7ce8504013b94949a1a0a78b23384259 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f543eefd16843778e940651a1560ae2" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 37945 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:17:53.042285 25771 mvcc.cc:205] Tried to move back new op lower bound from 7334510071967657984 to 7334510071898492928. Current Snapshot: MvccSnapshot[applied={T|T < 7334510071967657984}]
I20260929 03:17:53.042821 25772 mvcc.cc:205] Tried to move back new op lower bound from 7334510071967657984 to 7334510071898492928. Current Snapshot: MvccSnapshot[applied={T|T < 7334510071967657984 or (T in {7334510071967657984})}]
I20260929 03:17:53.043177 25773 mvcc.cc:205] Tried to move back new op lower bound from 7334510071967657984 to 7334510071898492928. Current Snapshot: MvccSnapshot[applied={T|T < 7334510071967657984}]
I20260929 03:17:53.045104 25287 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55336:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:17:53.045200 25287 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55336:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:17:53.046010 25287 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ebba074c6cca4697a4134f751768fb64: Sending DeleteTablet for 3 replicas of tablet e8abb85a646a4ec7b62926041ee8d894
I20260929 03:17:53.046416 25545 tablet_service.cc:1514] Processing DeleteTablet for tablet e8abb85a646a4ec7b62926041ee8d894 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:17:53 UTC) from {username='kudu', principal='kudu/127.20.178.126@KRBTEST.COM'} at 127.0.0.1:34742
I20260929 03:17:53.046422 25678 tablet_service.cc:1514] Processing DeleteTablet for tablet e8abb85a646a4ec7b62926041ee8d894 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:17:53 UTC) from {username='kudu', principal='kudu/127.20.178.126@KRBTEST.COM'} at 127.0.0.1:44730
I20260929 03:17:53.046414 25412 tablet_service.cc:1514] Processing DeleteTablet for tablet e8abb85a646a4ec7b62926041ee8d894 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:17:53 UTC) from {username='kudu', principal='kudu/127.20.178.126@KRBTEST.COM'} at 127.0.0.1:55942
I20260929 03:17:53.046656 25781 tablet_replica.cc:330] T e8abb85a646a4ec7b62926041ee8d894 P 1d36eeb42cff43788b9580f64e5df922: stopping tablet replica
I20260929 03:17:53.046754 25781 raft_consensus.cc:2229] T e8abb85a646a4ec7b62926041ee8d894 P 1d36eeb42cff43788b9580f64e5df922 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:17:53.046859 25781 raft_consensus.cc:2258] T e8abb85a646a4ec7b62926041ee8d894 P 1d36eeb42cff43788b9580f64e5df922 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:17:53.046849 21193 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuoaHbX/build/release/bin/kudu with pid 25350
I20260929 03:17:53.047039 25780 tablet_replica.cc:330] T e8abb85a646a4ec7b62926041ee8d894 P 7ce8504013b94949a1a0a78b23384259: stopping tablet replica
I20260929 03:17:53.047127 25780 raft_consensus.cc:2229] T e8abb85a646a4ec7b62926041ee8d894 P 7ce8504013b94949a1a0a78b23384259 [term 1 LEADER]: Raft consensus shutting down.
I20260929 03:17:53.047227 25780 raft_consensus.cc:2258] T e8abb85a646a4ec7b62926041ee8d894 P 7ce8504013b94949a1a0a78b23384259 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:17:53.047734 25780 ts_tablet_manager.cc:1905] T e8abb85a646a4ec7b62926041ee8d894 P 7ce8504013b94949a1a0a78b23384259: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:17:53.049527 25780 ts_tablet_manager.cc:1918] T e8abb85a646a4ec7b62926041ee8d894 P 7ce8504013b94949a1a0a78b23384259: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:17:53.049605 25780 log.cc:1196] T e8abb85a646a4ec7b62926041ee8d894 P 7ce8504013b94949a1a0a78b23384259: Deleting WAL directory at /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651852476208-21193-0/minicluster-data/ts-1/wal/wals/e8abb85a646a4ec7b62926041ee8d894
I20260929 03:17:53.049893 25780 ts_tablet_manager.cc:1939] T e8abb85a646a4ec7b62926041ee8d894 P 7ce8504013b94949a1a0a78b23384259: Deleting consensus metadata
I20260929 03:17:53.050325 25273 catalog_manager.cc:4841] TS 7ce8504013b94949a1a0a78b23384259 (127.20.178.66:43787): tablet e8abb85a646a4ec7b62926041ee8d894 (table test-table [id=46a2e9231be7476f87e31e07f99d266a]) successfully deleted
W20260929 03:17:53.052759 25272 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.20.178.65:42493 (error 108)
I20260929 03:17:53.053020 21193 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuoaHbX/build/release/bin/kudu with pid 25483
I20260929 03:17:53.053328 25754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:53.053207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.65:42493 (local address 127.0.0.1:55944)
0929 03:17:53.053257 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0929 03:17:53.053289 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.178.65:42493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260929 03:17:53.053417 25272 catalog_manager.cc:4584] TS 1d36eeb42cff43788b9580f64e5df922 (127.20.178.65:42493): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e8abb85a646a4ec7b62926041ee8d894: Network error: Client connection negotiation failed: client connection to 127.20.178.65:42493: connect: Connection refused (error 111)
I20260929 03:17:53.053622 25782 tablet_replica.cc:330] T e8abb85a646a4ec7b62926041ee8d894 P 2f543eefd16843778e940651a1560ae2: stopping tablet replica
I20260929 03:17:53.053743 25782 raft_consensus.cc:2229] T e8abb85a646a4ec7b62926041ee8d894 P 2f543eefd16843778e940651a1560ae2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:17:53.053854 25782 raft_consensus.cc:2258] T e8abb85a646a4ec7b62926041ee8d894 P 2f543eefd16843778e940651a1560ae2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:17:53.054230 25782 ts_tablet_manager.cc:1905] T e8abb85a646a4ec7b62926041ee8d894 P 2f543eefd16843778e940651a1560ae2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:17:53.055617 25782 ts_tablet_manager.cc:1918] T e8abb85a646a4ec7b62926041ee8d894 P 2f543eefd16843778e940651a1560ae2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:17:53.055701 25782 log.cc:1196] T e8abb85a646a4ec7b62926041ee8d894 P 2f543eefd16843778e940651a1560ae2: Deleting WAL directory at /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651852476208-21193-0/minicluster-data/ts-2/wal/wals/e8abb85a646a4ec7b62926041ee8d894
I20260929 03:17:53.055935 25782 ts_tablet_manager.cc:1939] T e8abb85a646a4ec7b62926041ee8d894 P 2f543eefd16843778e940651a1560ae2: Deleting consensus metadata
I20260929 03:17:53.056329 25275 catalog_manager.cc:4841] TS 2f543eefd16843778e940651a1560ae2 (127.20.178.67:37945): tablet e8abb85a646a4ec7b62926041ee8d894 (table test-table [id=46a2e9231be7476f87e31e07f99d266a]) successfully deleted
I20260929 03:17:53.058601 21193 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuoaHbX/build/release/bin/kudu with pid 25616
I20260929 03:17:53.065029 21193 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuoaHbX/build/release/bin/kudu with pid 25257
2026-09-29T03:17:53Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (1868 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-29T03:17:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:17:53Z Disabled control of system clock
I20260929 03:17:53.091887 21193 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuoaHbX/build/release/bin/kudu
/tmp/dist-test-taskuoaHbX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790651852476208-21193-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790651852476208-21193-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790651852476208-21193-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790651852476208-21193-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.178.126:44277
--webserver_interface=127.20.178.126
--webserver_port=0
--builtin_ntp_servers=127.20.178.84:39129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.178.126:44277
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260929 03:17:53.170465 25787 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:17:53.170696 25787 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:17:53.170717 25787 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:17:53.171772 25787 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:17:53.171806 25787 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:17:53.171819 25787 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:17:53.171832 25787 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:17:53.171844 25787 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:17:53.173116 25787 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.178.84:39129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790651852476208-21193-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790651852476208-21193-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.178.126:44277
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.178.126:44277
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790651852476208-21193-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.178.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25787
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790651852476208-21193-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:17:53.173308 25787 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:17:53.173540 25787 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:17:53.178139 25792 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:17:53.178118 25795 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:17:53.178324 25787 server_base.cc:1034] running on GCE node
W20260929 03:17:53.178483 25793 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:17:53.178785 25787 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:17:53.179033 25787 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:17:53.180171 25787 hybrid_clock.cc:648] HybridClock initialized: now 1790651873180160 us; error 28 us; skew 500 ppm
I20260929 03:17:53.181311 25787 webserver.cc:460] Webserver started at http://127.20.178.126:35691/ using document root <none> and password file <none>
I20260929 03:17:53.181504 25787 fs_manager.cc:359] Metadata directory not provided
I20260929 03:17:53.181543 25787 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:17:53.181629 25787 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:17:53.182598 25787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790651852476208-21193-0/minicluster-data/master-0/data/instance:
uuid: "2d1f8c54500b42df8fd4c64a3a62b369"
format_stamp: "Formatted at 2026-09-29 03:17:53 on dist-test-slave-c9qh"
server_key: "b930491bad96630a3c6c58bd0950d531"
server_key_iv: "037d41a8f770635b9e8ef5c85ee8ba12"
server_key_version: "encryptionkey@0"
I20260929 03:17:53.182897 25787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790651852476208-21193-0/minicluster-data/master-0/wal/instance:
uuid: "2d1f8c54500b42df8fd4c64a3a62b369"
format_stamp: "Formatted at 2026-09-29 03:17:53 on dist-test-slave-c9qh"
server_key: "b930491bad96630a3c6c58bd0950d531"
server_key_iv: "037d41a8f770635b9e8ef5c85ee8ba12"
server_key_version: "encryptionkey@0"
I20260929 03:17:53.184078 25787 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 03:17:53.184798 25801 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:17:53.184952 25787 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 03:17:53.185037 25787 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790651852476208-21193-0/minicluster-data/master-0/data,/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790651852476208-21193-0/minicluster-data/master-0/wal
uuid: "2d1f8c54500b42df8fd4c64a3a62b369"
format_stamp: "Formatted at 2026-09-29 03:17:53 on dist-test-slave-c9qh"
server_key: "b930491bad96630a3c6c58bd0950d531"
server_key_iv: "037d41a8f770635b9e8ef5c85ee8ba12"
server_key_version: "encryptionkey@0"
I20260929 03:17:53.185101 25787 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790651852476208-21193-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790651852476208-21193-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790651852476208-21193-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:17:53.188220 25787 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:17:53.188408 25787 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:17:53.188499 25787 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:17:53.191956 25787 rpc_server.cc:307] RPC server started. Bound to: 127.20.178.126:44277
I20260929 03:17:53.192013 25853 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.178.126:44277 every 8 connection(s)
I20260929 03:17:53.192310 25787 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790651852476208-21193-0/minicluster-data/master-0/data/info.pb
I20260929 03:17:53.192987 25854 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:17:53.195483 21193 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuoaHbX/build/release/bin/kudu as pid 25787
I20260929 03:17:53.195562 21193 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790651852476208-21193-0/minicluster-data/master-0/wal/instance
I20260929 03:17:53.195521 25854 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d1f8c54500b42df8fd4c64a3a62b369: Bootstrap starting.
I20260929 03:17:53.195773 21193 external_mini_cluster.cc:1428] Setting key 931a633187bc492016467297237aff1b
I20260929 03:17:53.196188 25854 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2d1f8c54500b42df8fd4c64a3a62b369: Neither blocks nor log segments found. Creating new log.
I20260929 03:17:53.196525 25854 log.cc:824] T 00000000000000000000000000000000 P 2d1f8c54500b42df8fd4c64a3a62b369: Log is configured to *not* fsync() on all Append() calls
I20260929 03:17:53.197263 25854 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d1f8c54500b42df8fd4c64a3a62b369: No bootstrap required, opened a new log
I20260929 03:17:53.198450 25857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:53.196261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59794 (local address 127.20.178.126:44277)
0929 03:17:53.196453 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:53.196461 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:53.196495 (+ 34us) server_negotiation.cc:407] Connection header received
0929 03:17:53.197049 (+ 554us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:53.197058 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:53.197108 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:53.197253 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:17:53.198010 (+ 757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:17:53.198014 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:17:53.198015 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:17:53.198020 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:17:53.198064 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:17:53.198078 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:17:53.198167 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0929 03:17:53.198198 (+ 31us) server_negotiation.cc:299] Negotiation successful
0929 03:17:53.198265 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":79,"threads_started":1}
I20260929 03:17:53.199092 25854 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2d1f8c54500b42df8fd4c64a3a62b369 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d1f8c54500b42df8fd4c64a3a62b369" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 44277 } }
I20260929 03:17:53.199236 25854 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2d1f8c54500b42df8fd4c64a3a62b369 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:17:53.199261 25854 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2d1f8c54500b42df8fd4c64a3a62b369 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d1f8c54500b42df8fd4c64a3a62b369, State: Initialized, Role: FOLLOWER
I20260929 03:17:53.199342 25854 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2d1f8c54500b42df8fd4c64a3a62b369 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d1f8c54500b42df8fd4c64a3a62b369" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 44277 } }
I20260929 03:17:53.199432 25854 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2d1f8c54500b42df8fd4c64a3a62b369 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:17:53.199474 25854 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2d1f8c54500b42df8fd4c64a3a62b369 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:17:53.199544 25854 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2d1f8c54500b42df8fd4c64a3a62b369 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:17:53.200374 25854 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2d1f8c54500b42df8fd4c64a3a62b369 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d1f8c54500b42df8fd4c64a3a62b369" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 44277 } }
I20260929 03:17:53.200487 25854 leader_election.cc:302] T 00000000000000000000000000000000 P 2d1f8c54500b42df8fd4c64a3a62b369 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2d1f8c54500b42df8fd4c64a3a62b369; no voters:
I20260929 03:17:53.200707 25854 leader_election.cc:288] T 00000000000000000000000000000000 P 2d1f8c54500b42df8fd4c64a3a62b369 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:17:53.200722 25859 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2d1f8c54500b42df8fd4c64a3a62b369 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:17:53.200912 25859 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2d1f8c54500b42df8fd4c64a3a62b369 [term 1 LEADER]: Becoming Leader. State: Replica: 2d1f8c54500b42df8fd4c64a3a62b369, State: Running, Role: LEADER
I20260929 03:17:53.200989 25854 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2d1f8c54500b42df8fd4c64a3a62b369 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:17:53.201009 25859 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2d1f8c54500b42df8fd4c64a3a62b369 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d1f8c54500b42df8fd4c64a3a62b369" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 44277 } }
I20260929 03:17:53.201355 25860 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d1f8c54500b42df8fd4c64a3a62b369 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2d1f8c54500b42df8fd4c64a3a62b369" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d1f8c54500b42df8fd4c64a3a62b369" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 44277 } } }
I20260929 03:17:53.201468 25860 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d1f8c54500b42df8fd4c64a3a62b369 [sys.catalog]: This master's current role is: LEADER
I20260929 03:17:53.201359 25861 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d1f8c54500b42df8fd4c64a3a62b369 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2d1f8c54500b42df8fd4c64a3a62b369. Latest consensus state: current_term: 1 leader_uuid: "2d1f8c54500b42df8fd4c64a3a62b369" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d1f8c54500b42df8fd4c64a3a62b369" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 44277 } } }
I20260929 03:17:53.201582 25861 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d1f8c54500b42df8fd4c64a3a62b369 [sys.catalog]: This master's current role is: LEADER
I20260929 03:17:53.202178 25865 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:17:53.202579 25865 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:17:53.204128 25865 catalog_manager.cc:1347] Generated new cluster ID: a5c3263bdb76461a869a3bcefa0f66fe
I20260929 03:17:53.204183 25865 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:17:53.222545 25865 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:17:53.222723 25865 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:17:53.228658 25865 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2d1f8c54500b42df8fd4c64a3a62b369: Generated new TSK 0
I20260929 03:17:53.228840 25865 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 03:17:53.236291 21193 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuoaHbX/build/release/bin/kudu
/tmp/dist-test-taskuoaHbX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790651852476208-21193-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790651852476208-21193-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790651852476208-21193-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790651852476208-21193-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.178.65:0
--local_ip_for_outbound_sockets=127.20.178.65
--webserver_interface=127.20.178.65
--webserver_port=0
--tserver_master_addrs=127.20.178.126:44277
--builtin_ntp_servers=127.20.178.84:39129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260929 03:17:53.316767 25878 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:17:53.316987 25878 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:17:53.317020 25878 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:17:53.318177 25878 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:17:53.318238 25878 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:17:53.318282 25878 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.178.65
I20260929 03:17:53.319635 25878 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.178.84:39129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790651852476208-21193-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790651852476208-21193-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.178.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790651852476208-21193-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.178.65
--webserver_port=0
--tserver_master_addrs=127.20.178.126:44277
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25878
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.178.65
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790651852476208-21193-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:17:53.319892 25878 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:17:53.320185 25878 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:17:53.326030 25884 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:17:53.326009 25883 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:17:53.326221 25878 server_base.cc:1034] running on GCE node
W20260929 03:17:53.326295 25886 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:17:53.326529 25878 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:17:53.326768 25878 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:17:53.327926 25878 hybrid_clock.cc:648] HybridClock initialized: now 1790651873327911 us; error 30 us; skew 500 ppm
I20260929 03:17:53.329205 25878 webserver.cc:460] Webserver started at http://127.20.178.65:45043/ using document root <none> and password file <none>
I20260929 03:17:53.329444 25878 fs_manager.cc:359] Metadata directory not provided
I20260929 03:17:53.329510 25878 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:17:53.329659 25878 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:17:53.330668 25878 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790651852476208-21193-0/minicluster-data/ts-0/data/instance:
uuid: "353419b6379b484c899500d282cf0b28"
format_stamp: "Formatted at 2026-09-29 03:17:53 on dist-test-slave-c9qh"
server_key: "c71498740f00411d854a7c9e3392e438"
server_key_iv: "8be902faf64968d53a02209d298589df"
server_key_version: "encryptionkey@0"
I20260929 03:17:53.331018 25878 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790651852476208-21193-0/minicluster-data/ts-0/wal/instance:
uuid: "353419b6379b484c899500d282cf0b28"
format_stamp: "Formatted at 2026-09-29 03:17:53 on dist-test-slave-c9qh"
server_key: "c71498740f00411d854a7c9e3392e438"
server_key_iv: "8be902faf64968d53a02209d298589df"
server_key_version: "encryptionkey@0"
I20260929 03:17:53.332293 25878 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260929 03:17:53.333154 25892 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:17:53.333334 25878 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260929 03:17:53.333443 25878 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790651852476208-21193-0/minicluster-data/ts-0/data,/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790651852476208-21193-0/minicluster-data/ts-0/wal
uuid: "353419b6379b484c899500d282cf0b28"
format_stamp: "Formatted at 2026-09-29 03:17:53 on dist-test-slave-c9qh"
server_key: "c71498740f00411d854a7c9e3392e438"
server_key_iv: "8be902faf64968d53a02209d298589df"
server_key_version: "encryptionkey@0"
I20260929 03:17:53.333513 25878 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790651852476208-21193-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790651852476208-21193-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790651852476208-21193-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:17:53.336702 25878 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:17:53.336964 25878 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:17:53.337083 25878 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:17:53.337303 25878 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:17:53.337627 25878 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:17:53.337682 25878 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:17:53.337721 25878 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:17:53.337749 25878 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:17:53.343128 25878 rpc_server.cc:307] RPC server started. Bound to: 127.20.178.65:36921
I20260929 03:17:53.343190 26005 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.178.65:36921 every 8 connection(s)
I20260929 03:17:53.343500 25878 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790651852476208-21193-0/minicluster-data/ts-0/data/info.pb
I20260929 03:17:53.346382 25857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:53.344210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.65:49649 (local address 127.20.178.126:44277)
0929 03:17:53.344288 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:53.344292 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:53.344479 (+ 187us) server_negotiation.cc:407] Connection header received
0929 03:17:53.344969 (+ 490us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:53.344970 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:53.344975 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:53.345026 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:17:53.346195 (+ 1169us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:17:53.346196 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:17:53.346196 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:17:53.346200 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:17:53.346214 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:17:53.346218 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:17:53.346273 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0929 03:17:53.346322 (+ 49us) server_negotiation.cc:299] Negotiation successful
0929 03:17:53.346333 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 03:17:53.346611 26008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:53.344170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:44277 (local address 127.20.178.65:49649)
0929 03:17:53.344363 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0929 03:17:53.344395 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0929 03:17:53.344885 (+ 490us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:17:53.345077 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:17:53.345080 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:17:53.345088 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:17:53.346032 (+ 944us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 03:17:53.346032 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:17:53.346060 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 03:17:53.346087 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:17:53.346274 (+ 187us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:17:53.346277 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:17:53.346279 (+ 2us) client_negotiation.cc:769] Sending connection context
0929 03:17:53.346328 (+ 49us) client_negotiation.cc:240] Negotiation successful
0929 03:17:53.346373 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":113,"threads_started":1}
I20260929 03:17:53.347173 26006 heartbeater.cc:347] Connected to a master server at 127.20.178.126:44277
I20260929 03:17:53.347249 26006 heartbeater.cc:464] Registering TS with master...
I20260929 03:17:53.347409 26006 heartbeater.cc:511] Master 127.20.178.126:44277 requested a full tablet report, sending...
I20260929 03:17:53.347904 25818 ts_manager.cc:194] Registered new tserver with Master: 353419b6379b484c899500d282cf0b28 (127.20.178.65:36921)
I20260929 03:17:53.351078 21193 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuoaHbX/build/release/bin/kudu as pid 25878
I20260929 03:17:53.351181 21193 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790651852476208-21193-0/minicluster-data/ts-0/wal/instance
I20260929 03:17:53.351359 21193 external_mini_cluster.cc:1428] Setting key ed3eb25e252a6b37af6056b419b8ce12
I20260929 03:17:53.352205 21193 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuoaHbX/build/release/bin/kudu
/tmp/dist-test-taskuoaHbX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790651852476208-21193-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790651852476208-21193-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790651852476208-21193-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790651852476208-21193-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.178.66:0
--local_ip_for_outbound_sockets=127.20.178.66
--webserver_interface=127.20.178.66
--webserver_port=0
--tserver_master_addrs=127.20.178.126:44277
--builtin_ntp_servers=127.20.178.84:39129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260929 03:17:53.430277 26009 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:17:53.430490 26009 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:17:53.430523 26009 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:17:53.432139 26009 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:17:53.432195 26009 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:17:53.432238 26009 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.178.66
I20260929 03:17:53.433594 26009 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.178.84:39129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790651852476208-21193-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790651852476208-21193-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.178.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790651852476208-21193-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.178.66
--webserver_port=0
--tserver_master_addrs=127.20.178.126:44277
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26009
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.178.66
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790651852476208-21193-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:17:53.433861 26009 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:17:53.434154 26009 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:17:53.439345 26014 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:17:53.439368 26017 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:17:53.439437 26009 server_base.cc:1034] running on GCE node
W20260929 03:17:53.439368 26015 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:17:53.439869 26009 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:17:53.440115 26009 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:17:53.441253 26009 hybrid_clock.cc:648] HybridClock initialized: now 1790651873441226 us; error 43 us; skew 500 ppm
I20260929 03:17:53.442381 26009 webserver.cc:460] Webserver started at http://127.20.178.66:40603/ using document root <none> and password file <none>
I20260929 03:17:53.442623 26009 fs_manager.cc:359] Metadata directory not provided
I20260929 03:17:53.442672 26009 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:17:53.442834 26009 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:17:53.443876 26009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790651852476208-21193-0/minicluster-data/ts-1/data/instance:
uuid: "6fe43fdf8e774b73add23ed362f72f0e"
format_stamp: "Formatted at 2026-09-29 03:17:53 on dist-test-slave-c9qh"
server_key: "323cb8126eb2a9c07dd022f901c41ad1"
server_key_iv: "2a2b47cbc30ad533169e2bb447a81e74"
server_key_version: "encryptionkey@0"
I20260929 03:17:53.444226 26009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790651852476208-21193-0/minicluster-data/ts-1/wal/instance:
uuid: "6fe43fdf8e774b73add23ed362f72f0e"
format_stamp: "Formatted at 2026-09-29 03:17:53 on dist-test-slave-c9qh"
server_key: "323cb8126eb2a9c07dd022f901c41ad1"
server_key_iv: "2a2b47cbc30ad533169e2bb447a81e74"
server_key_version: "encryptionkey@0"
I20260929 03:17:53.445574 26009 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 03:17:53.446308 26023 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:17:53.446491 26009 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 03:17:53.446573 26009 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790651852476208-21193-0/minicluster-data/ts-1/data,/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790651852476208-21193-0/minicluster-data/ts-1/wal
uuid: "6fe43fdf8e774b73add23ed362f72f0e"
format_stamp: "Formatted at 2026-09-29 03:17:53 on dist-test-slave-c9qh"
server_key: "323cb8126eb2a9c07dd022f901c41ad1"
server_key_iv: "2a2b47cbc30ad533169e2bb447a81e74"
server_key_version: "encryptionkey@0"
I20260929 03:17:53.446625 26009 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790651852476208-21193-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790651852476208-21193-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790651852476208-21193-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:17:53.449885 26009 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:17:53.450093 26009 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:17:53.450183 26009 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:17:53.450330 26009 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:17:53.450558 26009 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:17:53.450588 26009 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:17:53.450608 26009 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:17:53.450620 26009 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:17:53.456012 26009 rpc_server.cc:307] RPC server started. Bound to: 127.20.178.66:33421
I20260929 03:17:53.456126 26136 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.178.66:33421 every 8 connection(s)
I20260929 03:17:53.456353 26009 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790651852476208-21193-0/minicluster-data/ts-1/data/info.pb
I20260929 03:17:53.459096 25857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:53.457055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.66:44671 (local address 127.20.178.126:44277)
0929 03:17:53.457113 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:53.457116 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:53.457334 (+ 218us) server_negotiation.cc:407] Connection header received
0929 03:17:53.457757 (+ 423us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:53.457758 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:53.457763 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:53.457814 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:17:53.458936 (+ 1122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:17:53.458937 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:17:53.458937 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:17:53.458940 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:17:53.458949 (+ 9us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:17:53.458952 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:17:53.458989 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0929 03:17:53.459036 (+ 47us) server_negotiation.cc:299] Negotiation successful
0929 03:17:53.459046 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260929 03:17:53.459327 26139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:53.457045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:44277 (local address 127.20.178.66:44671)
0929 03:17:53.457231 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0929 03:17:53.457268 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 03:17:53.457673 (+ 405us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:17:53.457878 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:17:53.457881 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:17:53.457889 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:17:53.458812 (+ 923us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 03:17:53.458813 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:17:53.458838 (+ 25us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 03:17:53.458866 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:17:53.458990 (+ 124us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:17:53.458992 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:17:53.458994 (+ 2us) client_negotiation.cc:769] Sending connection context
0929 03:17:53.459043 (+ 49us) client_negotiation.cc:240] Negotiation successful
0929 03:17:53.459087 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":85,"threads_started":1}
I20260929 03:17:53.459848 26137 heartbeater.cc:347] Connected to a master server at 127.20.178.126:44277
I20260929 03:17:53.459947 26137 heartbeater.cc:464] Registering TS with master...
I20260929 03:17:53.460100 26137 heartbeater.cc:511] Master 127.20.178.126:44277 requested a full tablet report, sending...
I20260929 03:17:53.460481 25818 ts_manager.cc:194] Registered new tserver with Master: 6fe43fdf8e774b73add23ed362f72f0e (127.20.178.66:33421)
I20260929 03:17:53.466343 21193 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuoaHbX/build/release/bin/kudu as pid 26009
I20260929 03:17:53.466449 21193 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790651852476208-21193-0/minicluster-data/ts-1/wal/instance
I20260929 03:17:53.466614 21193 external_mini_cluster.cc:1428] Setting key 18169238449883ea57fa08d32bee30fb
I20260929 03:17:53.467504 21193 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuoaHbX/build/release/bin/kudu
/tmp/dist-test-taskuoaHbX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790651852476208-21193-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790651852476208-21193-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790651852476208-21193-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790651852476208-21193-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.178.67:0
--local_ip_for_outbound_sockets=127.20.178.67
--webserver_interface=127.20.178.67
--webserver_port=0
--tserver_master_addrs=127.20.178.126:44277
--builtin_ntp_servers=127.20.178.84:39129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260929 03:17:53.549326 26140 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:17:53.549513 26140 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:17:53.549536 26140 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:17:53.550794 26140 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:17:53.550832 26140 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:17:53.550858 26140 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.178.67
I20260929 03:17:53.552166 26140 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.178.84:39129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790651852476208-21193-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790651852476208-21193-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.178.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790651852476208-21193-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.178.67
--webserver_port=0
--tserver_master_addrs=127.20.178.126:44277
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26140
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.178.67
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790651852476208-21193-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:17:53.552424 26140 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:17:53.552785 26140 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:17:53.557674 26146 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:17:53.557665 26148 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:17:53.557664 26145 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:17:53.557914 26140 server_base.cc:1034] running on GCE node
I20260929 03:17:53.558102 26140 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:17:53.558300 26140 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:17:53.559430 26140 hybrid_clock.cc:648] HybridClock initialized: now 1790651873559415 us; error 31 us; skew 500 ppm
I20260929 03:17:53.560563 26140 webserver.cc:460] Webserver started at http://127.20.178.67:36463/ using document root <none> and password file <none>
I20260929 03:17:53.560829 26140 fs_manager.cc:359] Metadata directory not provided
I20260929 03:17:53.560899 26140 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:17:53.561043 26140 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:17:53.562063 26140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790651852476208-21193-0/minicluster-data/ts-2/data/instance:
uuid: "4b68dd5338df46918b707960ed075c7d"
format_stamp: "Formatted at 2026-09-29 03:17:53 on dist-test-slave-c9qh"
server_key: "255362b46427955a2c22c545900c229e"
server_key_iv: "047048b816433da787a315e1b9a6e614"
server_key_version: "encryptionkey@0"
I20260929 03:17:53.562435 26140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790651852476208-21193-0/minicluster-data/ts-2/wal/instance:
uuid: "4b68dd5338df46918b707960ed075c7d"
format_stamp: "Formatted at 2026-09-29 03:17:53 on dist-test-slave-c9qh"
server_key: "255362b46427955a2c22c545900c229e"
server_key_iv: "047048b816433da787a315e1b9a6e614"
server_key_version: "encryptionkey@0"
I20260929 03:17:53.563704 26140 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 03:17:53.564388 26154 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:17:53.564548 26140 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 03:17:53.564659 26140 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790651852476208-21193-0/minicluster-data/ts-2/data,/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790651852476208-21193-0/minicluster-data/ts-2/wal
uuid: "4b68dd5338df46918b707960ed075c7d"
format_stamp: "Formatted at 2026-09-29 03:17:53 on dist-test-slave-c9qh"
server_key: "255362b46427955a2c22c545900c229e"
server_key_iv: "047048b816433da787a315e1b9a6e614"
server_key_version: "encryptionkey@0"
I20260929 03:17:53.564745 26140 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790651852476208-21193-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790651852476208-21193-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790651852476208-21193-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:17:53.567903 26140 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:17:53.568141 26140 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:17:53.568290 26140 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:17:53.568490 26140 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:17:53.568828 26140 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:17:53.568881 26140 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:17:53.568920 26140 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:17:53.568946 26140 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:17:53.574599 26140 rpc_server.cc:307] RPC server started. Bound to: 127.20.178.67:34939
I20260929 03:17:53.574638 26267 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.178.67:34939 every 8 connection(s)
I20260929 03:17:53.574981 26140 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790651852476208-21193-0/minicluster-data/ts-2/data/info.pb
I20260929 03:17:53.577708 25857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:53.575614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.67:51637 (local address 127.20.178.126:44277)
0929 03:17:53.575670 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:53.575674 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:53.575881 (+ 207us) server_negotiation.cc:407] Connection header received
0929 03:17:53.576314 (+ 433us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:53.576314 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:53.576319 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:53.576376 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:17:53.577496 (+ 1120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:17:53.577498 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:17:53.577498 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:17:53.577503 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:17:53.577527 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:17:53.577531 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:17:53.577593 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0929 03:17:53.577640 (+ 47us) server_negotiation.cc:299] Negotiation successful
0929 03:17:53.577653 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260929 03:17:53.577872 26270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:53.575599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:44277 (local address 127.20.178.67:51637)
0929 03:17:53.575766 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0929 03:17:53.575803 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 03:17:53.576232 (+ 429us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:17:53.576421 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:17:53.576424 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:17:53.576431 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:17:53.577357 (+ 926us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 03:17:53.577357 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:17:53.577382 (+ 25us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 03:17:53.577407 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:17:53.577593 (+ 186us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:17:53.577595 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:17:53.577597 (+ 2us) client_negotiation.cc:769] Sending connection context
0929 03:17:53.577645 (+ 48us) client_negotiation.cc:240] Negotiation successful
0929 03:17:53.577691 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":84,"threads_started":1}
I20260929 03:17:53.578413 26268 heartbeater.cc:347] Connected to a master server at 127.20.178.126:44277
I20260929 03:17:53.578521 26268 heartbeater.cc:464] Registering TS with master...
I20260929 03:17:53.578679 26268 heartbeater.cc:511] Master 127.20.178.126:44277 requested a full tablet report, sending...
I20260929 03:17:53.579066 25818 ts_manager.cc:194] Registered new tserver with Master: 4b68dd5338df46918b707960ed075c7d (127.20.178.67:34939)
I20260929 03:17:53.582114 21193 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuoaHbX/build/release/bin/kudu as pid 26140
I20260929 03:17:53.582207 21193 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790651852476208-21193-0/minicluster-data/ts-2/wal/instance
I20260929 03:17:53.582346 21193 external_mini_cluster.cc:1428] Setting key 0f79489e4e0dbf700608ef6fba2608b4
I20260929 03:17:53.583009 21193 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 03:17:53.585604 25857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:53.584402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59796 (local address 127.20.178.126:44277)
0929 03:17:53.584471 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:53.584473 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:53.584593 (+ 120us) server_negotiation.cc:407] Connection header received
0929 03:17:53.584683 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:53.584684 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:53.584689 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:53.584749 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:17:53.585396 (+ 647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:17:53.585397 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:17:53.585397 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:17:53.585400 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:17:53.585410 (+ 10us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:17:53.585413 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:17:53.585454 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0929 03:17:53.585491 (+ 37us) server_negotiation.cc:299] Negotiation successful
0929 03:17:53.585501 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 03:17:53.587337 25818 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59796:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 03:17:53.588967 25818 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 03:17:53.592877 26276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:53.591022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.67:34939 (local address 127.0.0.1:48298)
0929 03:17:53.591124 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0929 03:17:53.591160 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0929 03:17:53.591208 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:17:53.591715 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:17:53.591716 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:17:53.591719 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:17:53.592374 (+ 655us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 03:17:53.592374 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:17:53.592380 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 03:17:53.592392 (+ 12us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:17:53.592702 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:17:53.592709 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:17:53.592712 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 03:17:53.592777 (+ 65us) client_negotiation.cc:240] Negotiation successful
0929 03:17:53.592789 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":80,"thread_start_us":61,"threads_started":1}
I20260929 03:17:53.592978 26279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:53.591142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.66:33421 (local address 127.0.0.1:43562)
0929 03:17:53.591342 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0929 03:17:53.591352 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 03:17:53.591388 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:17:53.591696 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:17:53.591702 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:17:53.591708 (+ 6us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:17:53.592320 (+ 612us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 03:17:53.592321 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:17:53.592333 (+ 12us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 03:17:53.592350 (+ 17us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:17:53.592850 (+ 500us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:17:53.592850 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:17:53.592851 (+ 1us) client_negotiation.cc:769] Sending connection context
0929 03:17:53.592907 (+ 56us) client_negotiation.cc:240] Negotiation successful
0929 03:17:53.592919 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":83,"threads_started":1}
I20260929 03:17:53.593084 26277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:53.591136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48298 (local address 127.20.178.67:34939)
0929 03:17:53.591581 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:53.591584 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:53.591593 (+ 9us) server_negotiation.cc:407] Connection header received
0929 03:17:53.591615 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:53.591617 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:53.591625 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:53.591697 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:17:53.592536 (+ 839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:17:53.592541 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:17:53.592542 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:17:53.592546 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:17:53.592590 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:17:53.592613 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:17:53.592690 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0929 03:17:53.592980 (+ 290us) server_negotiation.cc:299] Negotiation successful
0929 03:17:53.592996 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":416,"thread_start_us":85,"threads_started":1}
I20260929 03:17:53.593258 26280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:53.591369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43562 (local address 127.20.178.66:33421)
0929 03:17:53.591528 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:53.591533 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:53.591545 (+ 12us) server_negotiation.cc:407] Connection header received
0929 03:17:53.591571 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:53.591573 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:53.591584 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:53.591895 (+ 311us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:17:53.592436 (+ 541us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:17:53.592441 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:17:53.592442 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:17:53.592446 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:17:53.592476 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:17:53.592505 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:17:53.592591 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0929 03:17:53.593076 (+ 485us) server_negotiation.cc:299] Negotiation successful
0929 03:17:53.593094 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":96,"threads_started":1}
I20260929 03:17:53.593513 26278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:53.591233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.65:36921 (local address 127.0.0.1:52996)
0929 03:17:53.591438 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0929 03:17:53.591446 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0929 03:17:53.591483 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:17:53.592373 (+ 890us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:17:53.592374 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:17:53.592377 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:17:53.592854 (+ 477us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 03:17:53.592855 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:17:53.592859 (+ 4us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 03:17:53.592868 (+ 9us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:17:53.593393 (+ 525us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:17:53.593393 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:17:53.593394 (+ 1us) client_negotiation.cc:769] Sending connection context
0929 03:17:53.593435 (+ 41us) client_negotiation.cc:240] Negotiation successful
0929 03:17:53.593447 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":69,"threads_started":1}
I20260929 03:17:53.593744 26281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:53.591470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52996 (local address 127.20.178.65:36921)
0929 03:17:53.591632 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:53.591636 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:53.591647 (+ 11us) server_negotiation.cc:407] Connection header received
0929 03:17:53.591673 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:53.591676 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:53.591686 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:53.591765 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:17:53.593122 (+ 1357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:17:53.593129 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:17:53.593129 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:17:53.593133 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:17:53.593230 (+ 97us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:17:53.593256 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:17:53.593614 (+ 358us) server_negotiation.cc:1035] Waiting for connection context
0929 03:17:53.593646 (+ 32us) server_negotiation.cc:299] Negotiation successful
0929 03:17:53.593662 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":77,"threads_started":1}
I20260929 03:17:53.593698 26202 tablet_service.cc:1467] Processing CreateTablet for tablet fafc253354fa4d5ca31448e3fcd0d636 (DEFAULT_TABLE table=test-table [id=69da8c1a0a804f7b902500323c7a9076]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:17:53.594064 26202 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fafc253354fa4d5ca31448e3fcd0d636. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:17:53.593981 26071 tablet_service.cc:1467] Processing CreateTablet for tablet fafc253354fa4d5ca31448e3fcd0d636 (DEFAULT_TABLE table=test-table [id=69da8c1a0a804f7b902500323c7a9076]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:17:53.594309 26071 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fafc253354fa4d5ca31448e3fcd0d636. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:17:53.594410 25940 tablet_service.cc:1467] Processing CreateTablet for tablet fafc253354fa4d5ca31448e3fcd0d636 (DEFAULT_TABLE table=test-table [id=69da8c1a0a804f7b902500323c7a9076]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:17:53.594703 25940 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fafc253354fa4d5ca31448e3fcd0d636. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:17:53.596359 26282 tablet_bootstrap.cc:492] T fafc253354fa4d5ca31448e3fcd0d636 P 4b68dd5338df46918b707960ed075c7d: Bootstrap starting.
I20260929 03:17:53.597086 26282 tablet_bootstrap.cc:654] T fafc253354fa4d5ca31448e3fcd0d636 P 4b68dd5338df46918b707960ed075c7d: Neither blocks nor log segments found. Creating new log.
I20260929 03:17:53.597386 26282 log.cc:824] T fafc253354fa4d5ca31448e3fcd0d636 P 4b68dd5338df46918b707960ed075c7d: Log is configured to *not* fsync() on all Append() calls
I20260929 03:17:53.598111 26282 tablet_bootstrap.cc:492] T fafc253354fa4d5ca31448e3fcd0d636 P 4b68dd5338df46918b707960ed075c7d: No bootstrap required, opened a new log
I20260929 03:17:53.598189 26282 ts_tablet_manager.cc:1397] T fafc253354fa4d5ca31448e3fcd0d636 P 4b68dd5338df46918b707960ed075c7d: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260929 03:17:53.600154 26282 raft_consensus.cc:348] T fafc253354fa4d5ca31448e3fcd0d636 P 4b68dd5338df46918b707960ed075c7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b68dd5338df46918b707960ed075c7d" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 34939 } } peers { permanent_uuid: "6fe43fdf8e774b73add23ed362f72f0e" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 33421 } } peers { permanent_uuid: "353419b6379b484c899500d282cf0b28" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 36921 } }
I20260929 03:17:53.600310 26282 raft_consensus.cc:374] T fafc253354fa4d5ca31448e3fcd0d636 P 4b68dd5338df46918b707960ed075c7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:17:53.600353 26282 raft_consensus.cc:729] T fafc253354fa4d5ca31448e3fcd0d636 P 4b68dd5338df46918b707960ed075c7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b68dd5338df46918b707960ed075c7d, State: Initialized, Role: FOLLOWER
I20260929 03:17:53.600467 26282 consensus_queue.cc:260] T fafc253354fa4d5ca31448e3fcd0d636 P 4b68dd5338df46918b707960ed075c7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b68dd5338df46918b707960ed075c7d" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 34939 } } peers { permanent_uuid: "6fe43fdf8e774b73add23ed362f72f0e" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 33421 } } peers { permanent_uuid: "353419b6379b484c899500d282cf0b28" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 36921 } }
I20260929 03:17:53.600751 26282 ts_tablet_manager.cc:1428] T fafc253354fa4d5ca31448e3fcd0d636 P 4b68dd5338df46918b707960ed075c7d: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260929 03:17:53.600796 26268 heartbeater.cc:503] Master 127.20.178.126:44277 was elected leader, sending a full tablet report...
I20260929 03:17:53.600787 26284 tablet_bootstrap.cc:492] T fafc253354fa4d5ca31448e3fcd0d636 P 353419b6379b484c899500d282cf0b28: Bootstrap starting.
I20260929 03:17:53.600852 26285 tablet_bootstrap.cc:492] T fafc253354fa4d5ca31448e3fcd0d636 P 6fe43fdf8e774b73add23ed362f72f0e: Bootstrap starting.
I20260929 03:17:53.601485 26284 tablet_bootstrap.cc:654] T fafc253354fa4d5ca31448e3fcd0d636 P 353419b6379b484c899500d282cf0b28: Neither blocks nor log segments found. Creating new log.
I20260929 03:17:53.601676 26285 tablet_bootstrap.cc:654] T fafc253354fa4d5ca31448e3fcd0d636 P 6fe43fdf8e774b73add23ed362f72f0e: Neither blocks nor log segments found. Creating new log.
I20260929 03:17:53.601796 26284 log.cc:824] T fafc253354fa4d5ca31448e3fcd0d636 P 353419b6379b484c899500d282cf0b28: Log is configured to *not* fsync() on all Append() calls
I20260929 03:17:53.602061 26285 log.cc:824] T fafc253354fa4d5ca31448e3fcd0d636 P 6fe43fdf8e774b73add23ed362f72f0e: Log is configured to *not* fsync() on all Append() calls
I20260929 03:17:53.602632 26284 tablet_bootstrap.cc:492] T fafc253354fa4d5ca31448e3fcd0d636 P 353419b6379b484c899500d282cf0b28: No bootstrap required, opened a new log
I20260929 03:17:53.602707 26284 ts_tablet_manager.cc:1397] T fafc253354fa4d5ca31448e3fcd0d636 P 353419b6379b484c899500d282cf0b28: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 03:17:53.602811 26285 tablet_bootstrap.cc:492] T fafc253354fa4d5ca31448e3fcd0d636 P 6fe43fdf8e774b73add23ed362f72f0e: No bootstrap required, opened a new log
I20260929 03:17:53.602871 26285 ts_tablet_manager.cc:1397] T fafc253354fa4d5ca31448e3fcd0d636 P 6fe43fdf8e774b73add23ed362f72f0e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 03:17:53.605345 26285 raft_consensus.cc:348] T fafc253354fa4d5ca31448e3fcd0d636 P 6fe43fdf8e774b73add23ed362f72f0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b68dd5338df46918b707960ed075c7d" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 34939 } } peers { permanent_uuid: "6fe43fdf8e774b73add23ed362f72f0e" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 33421 } } peers { permanent_uuid: "353419b6379b484c899500d282cf0b28" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 36921 } }
I20260929 03:17:53.605320 26284 raft_consensus.cc:348] T fafc253354fa4d5ca31448e3fcd0d636 P 353419b6379b484c899500d282cf0b28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b68dd5338df46918b707960ed075c7d" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 34939 } } peers { permanent_uuid: "6fe43fdf8e774b73add23ed362f72f0e" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 33421 } } peers { permanent_uuid: "353419b6379b484c899500d282cf0b28" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 36921 } }
I20260929 03:17:53.605513 26284 raft_consensus.cc:374] T fafc253354fa4d5ca31448e3fcd0d636 P 353419b6379b484c899500d282cf0b28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:17:53.605513 26285 raft_consensus.cc:374] T fafc253354fa4d5ca31448e3fcd0d636 P 6fe43fdf8e774b73add23ed362f72f0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:17:53.605548 26285 raft_consensus.cc:729] T fafc253354fa4d5ca31448e3fcd0d636 P 6fe43fdf8e774b73add23ed362f72f0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6fe43fdf8e774b73add23ed362f72f0e, State: Initialized, Role: FOLLOWER
I20260929 03:17:53.605548 26284 raft_consensus.cc:729] T fafc253354fa4d5ca31448e3fcd0d636 P 353419b6379b484c899500d282cf0b28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 353419b6379b484c899500d282cf0b28, State: Initialized, Role: FOLLOWER
I20260929 03:17:53.605667 26284 consensus_queue.cc:260] T fafc253354fa4d5ca31448e3fcd0d636 P 353419b6379b484c899500d282cf0b28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b68dd5338df46918b707960ed075c7d" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 34939 } } peers { permanent_uuid: "6fe43fdf8e774b73add23ed362f72f0e" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 33421 } } peers { permanent_uuid: "353419b6379b484c899500d282cf0b28" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 36921 } }
I20260929 03:17:53.605669 26285 consensus_queue.cc:260] T fafc253354fa4d5ca31448e3fcd0d636 P 6fe43fdf8e774b73add23ed362f72f0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b68dd5338df46918b707960ed075c7d" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 34939 } } peers { permanent_uuid: "6fe43fdf8e774b73add23ed362f72f0e" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 33421 } } peers { permanent_uuid: "353419b6379b484c899500d282cf0b28" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 36921 } }
I20260929 03:17:53.605913 26285 ts_tablet_manager.cc:1428] T fafc253354fa4d5ca31448e3fcd0d636 P 6fe43fdf8e774b73add23ed362f72f0e: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260929 03:17:53.605912 26284 ts_tablet_manager.cc:1428] T fafc253354fa4d5ca31448e3fcd0d636 P 353419b6379b484c899500d282cf0b28: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260929 03:17:53.606027 26006 heartbeater.cc:503] Master 127.20.178.126:44277 was elected leader, sending a full tablet report...
I20260929 03:17:53.606314 26137 heartbeater.cc:503] Master 127.20.178.126:44277 was elected leader, sending a full tablet report...
W20260929 03:17:53.706867 26138 tablet.cc:2369] T fafc253354fa4d5ca31448e3fcd0d636 P 6fe43fdf8e774b73add23ed362f72f0e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 03:17:53.825512 26269 tablet.cc:2369] T fafc253354fa4d5ca31448e3fcd0d636 P 4b68dd5338df46918b707960ed075c7d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 03:17:53.844152 26007 tablet.cc:2369] T fafc253354fa4d5ca31448e3fcd0d636 P 353419b6379b484c899500d282cf0b28: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 03:17:53.856842 26286 raft_consensus.cc:482] T fafc253354fa4d5ca31448e3fcd0d636 P 4b68dd5338df46918b707960ed075c7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:17:53.856994 26286 raft_consensus.cc:504] T fafc253354fa4d5ca31448e3fcd0d636 P 4b68dd5338df46918b707960ed075c7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b68dd5338df46918b707960ed075c7d" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 34939 } } peers { permanent_uuid: "6fe43fdf8e774b73add23ed362f72f0e" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 33421 } } peers { permanent_uuid: "353419b6379b484c899500d282cf0b28" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 36921 } }
I20260929 03:17:53.857412 26286 leader_election.cc:288] T fafc253354fa4d5ca31448e3fcd0d636 P 4b68dd5338df46918b707960ed075c7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6fe43fdf8e774b73add23ed362f72f0e (127.20.178.66:33421), 353419b6379b484c899500d282cf0b28 (127.20.178.65:36921)
I20260929 03:17:53.864948 26280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:53.857648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.67:56013 (local address 127.20.178.66:33421)
0929 03:17:53.857697 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:53.857698 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:53.857814 (+ 116us) server_negotiation.cc:407] Connection header received
0929 03:17:53.857845 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:53.857846 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:53.857851 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:53.857908 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:17:53.864686 (+ 6778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:17:53.864687 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:17:53.864688 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:17:53.864692 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:17:53.864708 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:17:53.864713 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:17:53.864776 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0929 03:17:53.864863 (+ 87us) server_negotiation.cc:299] Negotiation successful
0929 03:17:53.864877 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260929 03:17:53.864948 26281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:53.857668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.67:42527 (local address 127.20.178.65:36921)
0929 03:17:53.857752 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:53.857753 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:53.857760 (+ 7us) server_negotiation.cc:407] Connection header received
0929 03:17:53.857785 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:53.857803 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:53.857810 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:53.857879 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:17:53.864673 (+ 6794us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:17:53.864675 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:17:53.864675 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:17:53.864681 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:17:53.864700 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:17:53.864705 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:17:53.864774 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0929 03:17:53.864859 (+ 85us) server_negotiation.cc:299] Negotiation successful
0929 03:17:53.864876 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 03:17:53.864964 26270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:53.857583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.65:36921 (local address 127.20.178.67:42527)
0929 03:17:53.857645 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0929 03:17:53.857658 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 03:17:53.857697 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:17:53.857920 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:17:53.857921 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:17:53.857924 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:17:53.864510 (+ 6586us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 03:17:53.864511 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:17:53.864520 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 03:17:53.864538 (+ 18us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:17:53.864794 (+ 256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:17:53.864794 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:17:53.864795 (+ 1us) client_negotiation.cc:769] Sending connection context
0929 03:17:53.864845 (+ 50us) client_negotiation.cc:240] Negotiation successful
0929 03:17:53.864865 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43,"mutex_wait_us":2}
I20260929 03:17:53.864984 26291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:53.857589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.66:33421 (local address 127.20.178.67:56013)
0929 03:17:53.857743 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0929 03:17:53.857752 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 03:17:53.857800 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:17:53.857920 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:17:53.857921 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:17:53.857935 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:17:53.864510 (+ 6575us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 03:17:53.864511 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:17:53.864522 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 03:17:53.864538 (+ 16us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:17:53.864789 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:17:53.864790 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:17:53.864791 (+ 1us) client_negotiation.cc:769] Sending connection context
0929 03:17:53.864847 (+ 56us) client_negotiation.cc:240] Negotiation successful
0929 03:17:53.864863 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"mutex_wait_us":12,"thread_start_us":86,"threads_started":1}
I20260929 03:17:53.865320 25960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fafc253354fa4d5ca31448e3fcd0d636" candidate_uuid: "4b68dd5338df46918b707960ed075c7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "353419b6379b484c899500d282cf0b28" is_pre_election: true
I20260929 03:17:53.865445 26091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fafc253354fa4d5ca31448e3fcd0d636" candidate_uuid: "4b68dd5338df46918b707960ed075c7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6fe43fdf8e774b73add23ed362f72f0e" is_pre_election: true
I20260929 03:17:53.865489 25960 raft_consensus.cc:2454] T fafc253354fa4d5ca31448e3fcd0d636 P 353419b6379b484c899500d282cf0b28 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4b68dd5338df46918b707960ed075c7d in term 0.
I20260929 03:17:53.865573 26091 raft_consensus.cc:2454] T fafc253354fa4d5ca31448e3fcd0d636 P 6fe43fdf8e774b73add23ed362f72f0e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4b68dd5338df46918b707960ed075c7d in term 0.
I20260929 03:17:53.865741 26157 leader_election.cc:302] T fafc253354fa4d5ca31448e3fcd0d636 P 4b68dd5338df46918b707960ed075c7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 353419b6379b484c899500d282cf0b28, 4b68dd5338df46918b707960ed075c7d; no voters:
I20260929 03:17:53.865883 26286 raft_consensus.cc:2781] T fafc253354fa4d5ca31448e3fcd0d636 P 4b68dd5338df46918b707960ed075c7d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 03:17:53.865947 26286 raft_consensus.cc:482] T fafc253354fa4d5ca31448e3fcd0d636 P 4b68dd5338df46918b707960ed075c7d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 03:17:53.865970 26286 raft_consensus.cc:3037] T fafc253354fa4d5ca31448e3fcd0d636 P 4b68dd5338df46918b707960ed075c7d [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:17:53.866637 26286 raft_consensus.cc:504] T fafc253354fa4d5ca31448e3fcd0d636 P 4b68dd5338df46918b707960ed075c7d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b68dd5338df46918b707960ed075c7d" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 34939 } } peers { permanent_uuid: "6fe43fdf8e774b73add23ed362f72f0e" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 33421 } } peers { permanent_uuid: "353419b6379b484c899500d282cf0b28" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 36921 } }
I20260929 03:17:53.866784 26286 leader_election.cc:288] T fafc253354fa4d5ca31448e3fcd0d636 P 4b68dd5338df46918b707960ed075c7d [CANDIDATE]: Term 1 election: Requested vote from peers 6fe43fdf8e774b73add23ed362f72f0e (127.20.178.66:33421), 353419b6379b484c899500d282cf0b28 (127.20.178.65:36921)
I20260929 03:17:53.866928 25960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fafc253354fa4d5ca31448e3fcd0d636" candidate_uuid: "4b68dd5338df46918b707960ed075c7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "353419b6379b484c899500d282cf0b28"
I20260929 03:17:53.867007 25960 raft_consensus.cc:3037] T fafc253354fa4d5ca31448e3fcd0d636 P 353419b6379b484c899500d282cf0b28 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:17:53.866959 26091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fafc253354fa4d5ca31448e3fcd0d636" candidate_uuid: "4b68dd5338df46918b707960ed075c7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6fe43fdf8e774b73add23ed362f72f0e"
I20260929 03:17:53.867062 26091 raft_consensus.cc:3037] T fafc253354fa4d5ca31448e3fcd0d636 P 6fe43fdf8e774b73add23ed362f72f0e [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:17:53.867741 25960 raft_consensus.cc:2454] T fafc253354fa4d5ca31448e3fcd0d636 P 353419b6379b484c899500d282cf0b28 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b68dd5338df46918b707960ed075c7d in term 1.
I20260929 03:17:53.867836 26091 raft_consensus.cc:2454] T fafc253354fa4d5ca31448e3fcd0d636 P 6fe43fdf8e774b73add23ed362f72f0e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b68dd5338df46918b707960ed075c7d in term 1.
I20260929 03:17:53.867913 26157 leader_election.cc:302] T fafc253354fa4d5ca31448e3fcd0d636 P 4b68dd5338df46918b707960ed075c7d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 353419b6379b484c899500d282cf0b28, 4b68dd5338df46918b707960ed075c7d; no voters:
I20260929 03:17:53.867993 26286 raft_consensus.cc:2781] T fafc253354fa4d5ca31448e3fcd0d636 P 4b68dd5338df46918b707960ed075c7d [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:17:53.868142 26286 raft_consensus.cc:686] T fafc253354fa4d5ca31448e3fcd0d636 P 4b68dd5338df46918b707960ed075c7d [term 1 LEADER]: Becoming Leader. State: Replica: 4b68dd5338df46918b707960ed075c7d, State: Running, Role: LEADER
I20260929 03:17:53.868239 26286 consensus_queue.cc:237] T fafc253354fa4d5ca31448e3fcd0d636 P 4b68dd5338df46918b707960ed075c7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b68dd5338df46918b707960ed075c7d" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 34939 } } peers { permanent_uuid: "6fe43fdf8e774b73add23ed362f72f0e" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 33421 } } peers { permanent_uuid: "353419b6379b484c899500d282cf0b28" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 36921 } }
I20260929 03:17:53.869081 25818 catalog_manager.cc:5495] T fafc253354fa4d5ca31448e3fcd0d636 P 4b68dd5338df46918b707960ed075c7d reported cstate change: term changed from 0 to 1, leader changed from <none> to 4b68dd5338df46918b707960ed075c7d (127.20.178.67). New cstate: current_term: 1 leader_uuid: "4b68dd5338df46918b707960ed075c7d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b68dd5338df46918b707960ed075c7d" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 34939 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6fe43fdf8e774b73add23ed362f72f0e" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 33421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "353419b6379b484c899500d282cf0b28" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 36921 } health_report { overall_health: UNKNOWN } } }
I20260929 03:17:53.877671 26277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:53.876713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48308 (local address 127.20.178.67:34939)
0929 03:17:53.876777 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:53.876779 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:53.876792 (+ 13us) server_negotiation.cc:407] Connection header received
0929 03:17:53.876898 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:53.876899 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:53.876909 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:53.876957 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:17:53.877430 (+ 473us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:17:53.877431 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:17:53.877431 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:17:53.877434 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:17:53.877447 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:17:53.877451 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:17:53.877496 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0929 03:17:53.877523 (+ 27us) server_negotiation.cc:299] Negotiation successful
0929 03:17:53.877534 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260929 03:17:53.879287 25960 raft_consensus.cc:1261] T fafc253354fa4d5ca31448e3fcd0d636 P 353419b6379b484c899500d282cf0b28 [term 1 FOLLOWER]: Refusing update from remote peer 4b68dd5338df46918b707960ed075c7d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:17:53.879289 26091 raft_consensus.cc:1261] T fafc253354fa4d5ca31448e3fcd0d636 P 6fe43fdf8e774b73add23ed362f72f0e [term 1 FOLLOWER]: Refusing update from remote peer 4b68dd5338df46918b707960ed075c7d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:17:53.879657 26292 consensus_queue.cc:1035] T fafc253354fa4d5ca31448e3fcd0d636 P 4b68dd5338df46918b707960ed075c7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6fe43fdf8e774b73add23ed362f72f0e" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 33421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:17:53.879886 26286 consensus_queue.cc:1035] T fafc253354fa4d5ca31448e3fcd0d636 P 4b68dd5338df46918b707960ed075c7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "353419b6379b484c899500d282cf0b28" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 36921 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:17:53.884007 26296 mvcc.cc:205] Tried to move back new op lower bound from 7334510075407024128 to 7334510075364851712. Current Snapshot: MvccSnapshot[applied={T|T < 7334510075407024128 or (T in {7334510075407024128})}]
I20260929 03:17:53.884155 25817 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:59796:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:17:53.884225 25817 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:59796:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:17:53.884975 25817 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2d1f8c54500b42df8fd4c64a3a62b369: Sending DeleteTablet for 3 replicas of tablet fafc253354fa4d5ca31448e3fcd0d636
I20260929 03:17:53.885286 26071 tablet_service.cc:1514] Processing DeleteTablet for tablet fafc253354fa4d5ca31448e3fcd0d636 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:17:53 UTC) from {username='slave'} at 127.0.0.1:43562
I20260929 03:17:53.885315 25940 tablet_service.cc:1514] Processing DeleteTablet for tablet fafc253354fa4d5ca31448e3fcd0d636 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:17:53 UTC) from {username='slave'} at 127.0.0.1:52996
I20260929 03:17:53.885500 26306 tablet_replica.cc:330] T fafc253354fa4d5ca31448e3fcd0d636 P 353419b6379b484c899500d282cf0b28: stopping tablet replica
I20260929 03:17:53.885594 26306 raft_consensus.cc:2229] T fafc253354fa4d5ca31448e3fcd0d636 P 353419b6379b484c899500d282cf0b28 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:17:53.885603 26202 tablet_service.cc:1514] Processing DeleteTablet for tablet fafc253354fa4d5ca31448e3fcd0d636 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:17:53 UTC) from {username='slave'} at 127.0.0.1:48298
I20260929 03:17:53.885717 26306 raft_consensus.cc:2258] T fafc253354fa4d5ca31448e3fcd0d636 P 353419b6379b484c899500d282cf0b28 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:17:53.885834 26307 tablet_replica.cc:330] T fafc253354fa4d5ca31448e3fcd0d636 P 4b68dd5338df46918b707960ed075c7d: stopping tablet replica
I20260929 03:17:53.885917 21193 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuoaHbX/build/release/bin/kudu with pid 25878
I20260929 03:17:53.886004 26307 raft_consensus.cc:2229] T fafc253354fa4d5ca31448e3fcd0d636 P 4b68dd5338df46918b707960ed075c7d [term 1 LEADER]: Raft consensus shutting down.
I20260929 03:17:53.886132 26307 raft_consensus.cc:2258] T fafc253354fa4d5ca31448e3fcd0d636 P 4b68dd5338df46918b707960ed075c7d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:17:53.886437 26305 tablet_replica.cc:330] T fafc253354fa4d5ca31448e3fcd0d636 P 6fe43fdf8e774b73add23ed362f72f0e: stopping tablet replica
I20260929 03:17:53.886531 26305 raft_consensus.cc:2229] T fafc253354fa4d5ca31448e3fcd0d636 P 6fe43fdf8e774b73add23ed362f72f0e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:17:53.886637 26305 raft_consensus.cc:2258] T fafc253354fa4d5ca31448e3fcd0d636 P 6fe43fdf8e774b73add23ed362f72f0e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:17:53.887128 26305 ts_tablet_manager.cc:1905] T fafc253354fa4d5ca31448e3fcd0d636 P 6fe43fdf8e774b73add23ed362f72f0e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:17:53.888135 26307 ts_tablet_manager.cc:1905] T fafc253354fa4d5ca31448e3fcd0d636 P 4b68dd5338df46918b707960ed075c7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:17:53.889220 26305 ts_tablet_manager.cc:1918] T fafc253354fa4d5ca31448e3fcd0d636 P 6fe43fdf8e774b73add23ed362f72f0e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:17:53.889297 26305 log.cc:1196] T fafc253354fa4d5ca31448e3fcd0d636 P 6fe43fdf8e774b73add23ed362f72f0e: Deleting WAL directory at /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790651852476208-21193-0/minicluster-data/ts-1/wal/wals/fafc253354fa4d5ca31448e3fcd0d636
I20260929 03:17:53.889547 26305 ts_tablet_manager.cc:1939] T fafc253354fa4d5ca31448e3fcd0d636 P 6fe43fdf8e774b73add23ed362f72f0e: Deleting consensus metadata
I20260929 03:17:53.890065 26307 ts_tablet_manager.cc:1918] T fafc253354fa4d5ca31448e3fcd0d636 P 4b68dd5338df46918b707960ed075c7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:17:53.890069 25803 catalog_manager.cc:4841] TS 6fe43fdf8e774b73add23ed362f72f0e (127.20.178.66:33421): tablet fafc253354fa4d5ca31448e3fcd0d636 (table test-table [id=69da8c1a0a804f7b902500323c7a9076]) successfully deleted
I20260929 03:17:53.890205 26307 log.cc:1196] T fafc253354fa4d5ca31448e3fcd0d636 P 4b68dd5338df46918b707960ed075c7d: Deleting WAL directory at /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790651852476208-21193-0/minicluster-data/ts-2/wal/wals/fafc253354fa4d5ca31448e3fcd0d636
I20260929 03:17:53.890491 26307 ts_tablet_manager.cc:1939] T fafc253354fa4d5ca31448e3fcd0d636 P 4b68dd5338df46918b707960ed075c7d: Deleting consensus metadata
I20260929 03:17:53.890870 25804 catalog_manager.cc:4841] TS 4b68dd5338df46918b707960ed075c7d (127.20.178.67:34939): tablet fafc253354fa4d5ca31448e3fcd0d636 (table test-table [id=69da8c1a0a804f7b902500323c7a9076]) successfully deleted
W20260929 03:17:53.893086 25804 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.20.178.65:36921 (error 108)
I20260929 03:17:53.893286 21193 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuoaHbX/build/release/bin/kudu with pid 26009
I20260929 03:17:53.894014 26278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:53.893441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.65:36921 (local address 127.0.0.1:53006)
0929 03:17:53.893930 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0929 03:17:53.893972 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.178.65:36921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":467}
W20260929 03:17:53.894981 25804 catalog_manager.cc:4584] TS 353419b6379b484c899500d282cf0b28 (127.20.178.65:36921): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet fafc253354fa4d5ca31448e3fcd0d636: Network error: Client connection negotiation failed: client connection to 127.20.178.65:36921: connect: Connection refused (error 111)
I20260929 03:17:53.898283 21193 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuoaHbX/build/release/bin/kudu with pid 26140
I20260929 03:17:53.902704 21193 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuoaHbX/build/release/bin/kudu with pid 25787
2026-09-29T03:17:53Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (836 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260929 03:17:53.922030 21193 mini_oidc.cc:152] Starting JWKS server
I20260929 03:17:53.923050 21193 webserver.cc:460] Webserver started at https://127.0.0.1:40789/ using document root <none> and password file <none>
I20260929 03:17:53.923154 21193 mini_oidc.cc:189] Starting OIDC Discovery server
I20260929 03:17:53.923323 21193 webserver.cc:460] Webserver started at http://127.0.0.1:40075/ using document root <none> and password file <none>
2026-09-29T03:17:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:17:53Z Disabled control of system clock
I20260929 03:17:53.931214 21193 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuoaHbX/build/release/bin/kudu
/tmp/dist-test-taskuoaHbX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790651852476208-21193-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790651852476208-21193-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790651852476208-21193-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790651852476208-21193-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.178.126:41843
--webserver_interface=127.20.178.126
--webserver_port=0
--builtin_ntp_servers=127.20.178.84:46489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.178.126:41843
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40789/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790651852476208-21193-0/testchainca.cert with env {}
W20260929 03:17:54.009917 26314 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:17:54.010111 26314 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:17:54.010133 26314 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:17:54.011437 26314 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:17:54.011483 26314 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:17:54.011497 26314 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:17:54.011550 26314 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:17:54.011579 26314 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 03:17:54.011595 26314 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 03:17:54.011618 26314 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:40789/jwks.json
I20260929 03:17:54.013240 26314 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.178.84:46489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790651852476208-21193-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790651852476208-21193-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.178.126:41843
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.178.126:41843
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40789/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790651852476208-21193-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.178.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790651852476208-21193-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26314
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790651852476208-21193-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:17:54.013494 26314 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:17:54.013717 26314 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:17:54.018458 26320 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:17:54.018447 26322 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:17:54.018447 26319 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:17:54.018565 26314 server_base.cc:1034] running on GCE node
I20260929 03:17:54.018887 26314 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:17:54.019179 26314 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:17:54.020346 26314 hybrid_clock.cc:648] HybridClock initialized: now 1790651874020339 us; error 37 us; skew 500 ppm
I20260929 03:17:54.021574 26314 webserver.cc:460] Webserver started at http://127.20.178.126:41591/ using document root <none> and password file <none>
I20260929 03:17:54.021834 26314 fs_manager.cc:359] Metadata directory not provided
I20260929 03:17:54.021904 26314 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:17:54.022050 26314 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:17:54.023124 26314 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790651852476208-21193-0/minicluster-data/master-0/data/instance:
uuid: "ae989ce24e694e50b7973c0b0183347b"
format_stamp: "Formatted at 2026-09-29 03:17:54 on dist-test-slave-c9qh"
server_key: "ee0abec762844ab51c27068f1f733da6"
server_key_iv: "29c9c78539f8ac52791434547df25128"
server_key_version: "encryptionkey@0"
I20260929 03:17:54.023468 26314 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790651852476208-21193-0/minicluster-data/master-0/wal/instance:
uuid: "ae989ce24e694e50b7973c0b0183347b"
format_stamp: "Formatted at 2026-09-29 03:17:54 on dist-test-slave-c9qh"
server_key: "ee0abec762844ab51c27068f1f733da6"
server_key_iv: "29c9c78539f8ac52791434547df25128"
server_key_version: "encryptionkey@0"
I20260929 03:17:54.024889 26314 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 03:17:54.025600 26328 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:17:54.025768 26314 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 03:17:54.025867 26314 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790651852476208-21193-0/minicluster-data/master-0/data,/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790651852476208-21193-0/minicluster-data/master-0/wal
uuid: "ae989ce24e694e50b7973c0b0183347b"
format_stamp: "Formatted at 2026-09-29 03:17:54 on dist-test-slave-c9qh"
server_key: "ee0abec762844ab51c27068f1f733da6"
server_key_iv: "29c9c78539f8ac52791434547df25128"
server_key_version: "encryptionkey@0"
I20260929 03:17:54.025944 26314 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790651852476208-21193-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790651852476208-21193-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790651852476208-21193-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:17:54.039125 26314 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:17:54.039394 26314 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:17:54.039535 26314 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:17:54.043321 26314 rpc_server.cc:307] RPC server started. Bound to: 127.20.178.126:41843
I20260929 03:17:54.043346 26380 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.178.126:41843 every 8 connection(s)
I20260929 03:17:54.043663 26314 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790651852476208-21193-0/minicluster-data/master-0/data/info.pb
I20260929 03:17:54.044615 26381 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:17:54.044835 21193 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuoaHbX/build/release/bin/kudu as pid 26314
I20260929 03:17:54.044953 21193 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790651852476208-21193-0/minicluster-data/master-0/wal/instance
I20260929 03:17:54.045120 21193 external_mini_cluster.cc:1428] Setting key c42094ed48ae609f360d2ca53559178c
I20260929 03:17:54.048070 26381 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae989ce24e694e50b7973c0b0183347b: Bootstrap starting.
I20260929 03:17:54.048743 26381 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ae989ce24e694e50b7973c0b0183347b: Neither blocks nor log segments found. Creating new log.
I20260929 03:17:54.049023 26381 log.cc:824] T 00000000000000000000000000000000 P ae989ce24e694e50b7973c0b0183347b: Log is configured to *not* fsync() on all Append() calls
I20260929 03:17:54.049686 26381 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae989ce24e694e50b7973c0b0183347b: No bootstrap required, opened a new log
I20260929 03:17:54.050879 26384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:54.045565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48130 (local address 127.20.178.126:41843)
0929 03:17:54.045768 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:54.045777 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:54.045808 (+ 31us) server_negotiation.cc:407] Connection header received
0929 03:17:54.046307 (+ 499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:54.046315 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:54.046395 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:54.046547 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:17:54.046710 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:54.048049 (+ 1339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:54.048946 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:54.049428 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:54.049574 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:54.050242 (+ 668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:17:54.050246 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:17:54.050246 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:17:54.050423 (+ 177us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:17:54.050454 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:17:54.050465 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:17:54.050540 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0929 03:17:54.050615 (+ 75us) server_negotiation.cc:299] Negotiation successful
0929 03:17:54.050692 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":127,"thread_start_us":78,"threads_started":1}
I20260929 03:17:54.051843 21193 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 03:17:54.051971 26381 raft_consensus.cc:348] T 00000000000000000000000000000000 P ae989ce24e694e50b7973c0b0183347b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae989ce24e694e50b7973c0b0183347b" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 41843 } }
I20260929 03:17:54.052129 26381 raft_consensus.cc:374] T 00000000000000000000000000000000 P ae989ce24e694e50b7973c0b0183347b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:17:54.052174 26381 raft_consensus.cc:729] T 00000000000000000000000000000000 P ae989ce24e694e50b7973c0b0183347b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae989ce24e694e50b7973c0b0183347b, State: Initialized, Role: FOLLOWER
I20260929 03:17:54.052292 26381 consensus_queue.cc:260] T 00000000000000000000000000000000 P ae989ce24e694e50b7973c0b0183347b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae989ce24e694e50b7973c0b0183347b" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 41843 } }
I20260929 03:17:54.052371 26381 raft_consensus.cc:388] T 00000000000000000000000000000000 P ae989ce24e694e50b7973c0b0183347b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:17:54.052416 26381 raft_consensus.cc:482] T 00000000000000000000000000000000 P ae989ce24e694e50b7973c0b0183347b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:17:54.052462 26381 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ae989ce24e694e50b7973c0b0183347b [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:17:54.053114 26381 raft_consensus.cc:504] T 00000000000000000000000000000000 P ae989ce24e694e50b7973c0b0183347b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae989ce24e694e50b7973c0b0183347b" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 41843 } }
I20260929 03:17:54.053236 26381 leader_election.cc:302] T 00000000000000000000000000000000 P ae989ce24e694e50b7973c0b0183347b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ae989ce24e694e50b7973c0b0183347b; no voters:
I20260929 03:17:54.053395 26381 leader_election.cc:288] T 00000000000000000000000000000000 P ae989ce24e694e50b7973c0b0183347b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:17:54.053442 26386 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ae989ce24e694e50b7973c0b0183347b [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:17:54.053591 26386 raft_consensus.cc:686] T 00000000000000000000000000000000 P ae989ce24e694e50b7973c0b0183347b [term 1 LEADER]: Becoming Leader. State: Replica: ae989ce24e694e50b7973c0b0183347b, State: Running, Role: LEADER
I20260929 03:17:54.053668 26381 sys_catalog.cc:564] T 00000000000000000000000000000000 P ae989ce24e694e50b7973c0b0183347b [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:17:54.053674 26386 consensus_queue.cc:237] T 00000000000000000000000000000000 P ae989ce24e694e50b7973c0b0183347b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae989ce24e694e50b7973c0b0183347b" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 41843 } }
I20260929 03:17:54.054075 26388 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae989ce24e694e50b7973c0b0183347b [sys.catalog]: SysCatalogTable state changed. Reason: New leader ae989ce24e694e50b7973c0b0183347b. Latest consensus state: current_term: 1 leader_uuid: "ae989ce24e694e50b7973c0b0183347b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae989ce24e694e50b7973c0b0183347b" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 41843 } } }
I20260929 03:17:54.054076 26387 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae989ce24e694e50b7973c0b0183347b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ae989ce24e694e50b7973c0b0183347b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae989ce24e694e50b7973c0b0183347b" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 41843 } } }
I20260929 03:17:54.054298 26387 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae989ce24e694e50b7973c0b0183347b [sys.catalog]: This master's current role is: LEADER
I20260929 03:17:54.054479 26395 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:17:54.054240 26388 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae989ce24e694e50b7973c0b0183347b [sys.catalog]: This master's current role is: LEADER
I20260929 03:17:54.054989 26395 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:17:54.056474 26395 catalog_manager.cc:1347] Generated new cluster ID: d10827b7f5b74e229658d4dd1e24bf29
I20260929 03:17:54.056526 26395 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:17:54.064997 26395 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:17:54.065506 26395 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:17:54.069336 26395 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ae989ce24e694e50b7973c0b0183347b: Generated new TSK 0
I20260929 03:17:54.069499 26395 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260929 03:17:54.090372 26411 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260929 03:17:54.090472 26411 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260929 03:17:54.090740 26384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:54.075396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48140 (local address 127.20.178.126:41843)
0929 03:17:54.075458 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:54.075460 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:54.075607 (+ 147us) server_negotiation.cc:407] Connection header received
0929 03:17:54.075703 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:54.075703 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:54.075714 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:54.075778 (+ 64us) server_negotiation.cc:226] Negotiated authn=JWT
0929 03:17:54.075935 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:54.076393 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:54.077591 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:54.077799 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:54.077878 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:54.078091 (+ 213us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 03:17:54.090533 (+ 12442us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40789/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0929 03:17:54.090692 (+ 159us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:48140: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40789/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":39}
W20260929 03:17:54.090782 26384 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:48140: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40789/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260929 03:17:54.091243 26410 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:17:54.075340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:41843 (local address 127.0.0.1:48140)
0929 03:17:54.075524 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0929 03:17:54.075545 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 03:17:54.075616 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:17:54.075795 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:17:54.075795 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:17:54.075797 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0929 03:17:54.075873 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:54.075875 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:54.076480 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:54.076481 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:54.077479 (+ 998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:54.077482 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:54.077892 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:54.077893 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:54.077958 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:54.077979 (+ 21us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 03:17:54.090925 (+ 12946us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40789/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0929 03:17:54.091017 (+ 92us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.20.178.126:41843: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40789/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":81,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.20.178.126:41843: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40789/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260929 03:17:54.091763 21193 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuoaHbX/build/release/bin/kudu with pid 26314
2026-09-29T03:17:54Z chronyd exiting
I20260929 03:17:54.127022 21193 test_util.cc:175] -----------------------------------------------
I20260929 03:17:54.127090 21193 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790651852476208-21193-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (205 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260929 03:17:54.127910 21193 mini_oidc.cc:152] Starting JWKS server
I20260929 03:17:54.128593 21193 webserver.cc:460] Webserver started at https://127.0.0.1:43559/ using document root <none> and password file <none>
I20260929 03:17:54.128700 21193 mini_oidc.cc:189] Starting OIDC Discovery server
I20260929 03:17:54.128911 21193 webserver.cc:460] Webserver started at http://127.0.0.1:44225/ using document root <none> and password file <none>
2026-09-29T03:17:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:17:54Z Disabled control of system clock
I20260929 03:17:54.136142 21193 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuoaHbX/build/release/bin/kudu
/tmp/dist-test-taskuoaHbX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790651852476208-21193-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790651852476208-21193-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790651852476208-21193-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790651852476208-21193-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.178.126:43457
--webserver_interface=127.20.178.126
--webserver_port=0
--builtin_ntp_servers=127.20.178.84:42577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.178.126:43457
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43559/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790651852476208-21193-0/testchainca.cert with env {}
W20260929 03:17:54.213234 26417 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:17:54.213418 26417 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:17:54.213439 26417 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:17:54.214496 26417 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:17:54.214529 26417 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:17:54.214541 26417 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:17:54.214553 26417 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:17:54.214565 26417 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 03:17:54.214577 26417 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 03:17:54.214589 26417 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:43559/jwks.json
I20260929 03:17:54.215982 26417 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.178.84:42577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790651852476208-21193-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790651852476208-21193-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.178.126:43457
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.178.126:43457
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43559/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790651852476208-21193-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.178.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790651852476208-21193-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26417
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790651852476208-21193-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:17:54.216161 26417 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:17:54.216303 26417 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:17:54.220906 26425 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:17:54.221012 26417 server_base.cc:1034] running on GCE node
W20260929 03:17:54.220898 26422 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:17:54.221019 26423 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:17:54.221417 26417 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:17:54.221720 26417 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:17:54.222887 26417 hybrid_clock.cc:648] HybridClock initialized: now 1790651874222874 us; error 30 us; skew 500 ppm
I20260929 03:17:54.224150 26417 webserver.cc:460] Webserver started at http://127.20.178.126:46147/ using document root <none> and password file <none>
I20260929 03:17:54.224395 26417 fs_manager.cc:359] Metadata directory not provided
I20260929 03:17:54.224466 26417 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:17:54.224618 26417 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:17:54.225704 26417 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790651852476208-21193-0/minicluster-data/master-0/data/instance:
uuid: "e7404f5acacf4a0a9ba7a6228b78ee2d"
format_stamp: "Formatted at 2026-09-29 03:17:54 on dist-test-slave-c9qh"
server_key: "41dddc7cca2ffddfaa00e67df0df000f"
server_key_iv: "91e04f0d7b8e4f9456b6a2f19acbcabe"
server_key_version: "encryptionkey@0"
I20260929 03:17:54.226056 26417 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790651852476208-21193-0/minicluster-data/master-0/wal/instance:
uuid: "e7404f5acacf4a0a9ba7a6228b78ee2d"
format_stamp: "Formatted at 2026-09-29 03:17:54 on dist-test-slave-c9qh"
server_key: "41dddc7cca2ffddfaa00e67df0df000f"
server_key_iv: "91e04f0d7b8e4f9456b6a2f19acbcabe"
server_key_version: "encryptionkey@0"
I20260929 03:17:54.227259 26417 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260929 03:17:54.227945 26431 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:17:54.228121 26417 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 03:17:54.228217 26417 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790651852476208-21193-0/minicluster-data/master-0/data,/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790651852476208-21193-0/minicluster-data/master-0/wal
uuid: "e7404f5acacf4a0a9ba7a6228b78ee2d"
format_stamp: "Formatted at 2026-09-29 03:17:54 on dist-test-slave-c9qh"
server_key: "41dddc7cca2ffddfaa00e67df0df000f"
server_key_iv: "91e04f0d7b8e4f9456b6a2f19acbcabe"
server_key_version: "encryptionkey@0"
I20260929 03:17:54.228300 26417 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790651852476208-21193-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790651852476208-21193-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790651852476208-21193-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:17:54.250801 26417 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:17:54.251096 26417 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:17:54.251255 26417 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:17:54.255087 26417 rpc_server.cc:307] RPC server started. Bound to: 127.20.178.126:43457
I20260929 03:17:54.255120 26483 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.178.126:43457 every 8 connection(s)
I20260929 03:17:54.255443 26417 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790651852476208-21193-0/minicluster-data/master-0/data/info.pb
I20260929 03:17:54.256117 26484 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:17:54.258862 26484 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7404f5acacf4a0a9ba7a6228b78ee2d: Bootstrap starting.
I20260929 03:17:54.259438 26484 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e7404f5acacf4a0a9ba7a6228b78ee2d: Neither blocks nor log segments found. Creating new log.
I20260929 03:17:54.259734 26484 log.cc:824] T 00000000000000000000000000000000 P e7404f5acacf4a0a9ba7a6228b78ee2d: Log is configured to *not* fsync() on all Append() calls
I20260929 03:17:54.259737 21193 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuoaHbX/build/release/bin/kudu as pid 26417
I20260929 03:17:54.259850 21193 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790651852476208-21193-0/minicluster-data/master-0/wal/instance
I20260929 03:17:54.260035 21193 external_mini_cluster.cc:1428] Setting key 6bf7f656e005d7f5802acc57daf52a25
I20260929 03:17:54.260520 26484 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7404f5acacf4a0a9ba7a6228b78ee2d: No bootstrap required, opened a new log
I20260929 03:17:54.262338 26484 raft_consensus.cc:348] T 00000000000000000000000000000000 P e7404f5acacf4a0a9ba7a6228b78ee2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7404f5acacf4a0a9ba7a6228b78ee2d" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 43457 } }
I20260929 03:17:54.262470 26484 raft_consensus.cc:374] T 00000000000000000000000000000000 P e7404f5acacf4a0a9ba7a6228b78ee2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:17:54.262491 26484 raft_consensus.cc:729] T 00000000000000000000000000000000 P e7404f5acacf4a0a9ba7a6228b78ee2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7404f5acacf4a0a9ba7a6228b78ee2d, State: Initialized, Role: FOLLOWER
I20260929 03:17:54.262655 26484 consensus_queue.cc:260] T 00000000000000000000000000000000 P e7404f5acacf4a0a9ba7a6228b78ee2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7404f5acacf4a0a9ba7a6228b78ee2d" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 43457 } }
I20260929 03:17:54.262741 26484 raft_consensus.cc:388] T 00000000000000000000000000000000 P e7404f5acacf4a0a9ba7a6228b78ee2d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:17:54.262792 26484 raft_consensus.cc:482] T 00000000000000000000000000000000 P e7404f5acacf4a0a9ba7a6228b78ee2d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:17:54.262838 26484 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e7404f5acacf4a0a9ba7a6228b78ee2d [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:17:54.263367 26484 raft_consensus.cc:504] T 00000000000000000000000000000000 P e7404f5acacf4a0a9ba7a6228b78ee2d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7404f5acacf4a0a9ba7a6228b78ee2d" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 43457 } }
I20260929 03:17:54.263454 26484 leader_election.cc:302] T 00000000000000000000000000000000 P e7404f5acacf4a0a9ba7a6228b78ee2d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e7404f5acacf4a0a9ba7a6228b78ee2d; no voters:
I20260929 03:17:54.263628 26484 leader_election.cc:288] T 00000000000000000000000000000000 P e7404f5acacf4a0a9ba7a6228b78ee2d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:17:54.263648 26489 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e7404f5acacf4a0a9ba7a6228b78ee2d [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:17:54.263839 26489 raft_consensus.cc:686] T 00000000000000000000000000000000 P e7404f5acacf4a0a9ba7a6228b78ee2d [term 1 LEADER]: Becoming Leader. State: Replica: e7404f5acacf4a0a9ba7a6228b78ee2d, State: Running, Role: LEADER
I20260929 03:17:54.263927 26484 sys_catalog.cc:564] T 00000000000000000000000000000000 P e7404f5acacf4a0a9ba7a6228b78ee2d [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:17:54.263935 26489 consensus_queue.cc:237] T 00000000000000000000000000000000 P e7404f5acacf4a0a9ba7a6228b78ee2d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7404f5acacf4a0a9ba7a6228b78ee2d" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 43457 } }
I20260929 03:17:54.264248 26490 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7404f5acacf4a0a9ba7a6228b78ee2d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e7404f5acacf4a0a9ba7a6228b78ee2d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7404f5acacf4a0a9ba7a6228b78ee2d" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 43457 } } }
I20260929 03:17:54.264354 26490 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7404f5acacf4a0a9ba7a6228b78ee2d [sys.catalog]: This master's current role is: LEADER
I20260929 03:17:54.264247 26491 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7404f5acacf4a0a9ba7a6228b78ee2d [sys.catalog]: SysCatalogTable state changed. Reason: New leader e7404f5acacf4a0a9ba7a6228b78ee2d. Latest consensus state: current_term: 1 leader_uuid: "e7404f5acacf4a0a9ba7a6228b78ee2d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7404f5acacf4a0a9ba7a6228b78ee2d" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 43457 } } }
I20260929 03:17:54.264521 26491 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7404f5acacf4a0a9ba7a6228b78ee2d [sys.catalog]: This master's current role is: LEADER
I20260929 03:17:54.264722 26496 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:17:54.265501 26496 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:17:54.267179 26496 catalog_manager.cc:1347] Generated new cluster ID: 0c8acb59d2ee4c78a26abc2a4d3932e8
I20260929 03:17:54.267236 26496 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:17:54.267719 26488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:54.260510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50058 (local address 127.20.178.126:43457)
0929 03:17:54.260762 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:54.260771 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:54.260808 (+ 37us) server_negotiation.cc:407] Connection header received
0929 03:17:54.262325 (+ 1517us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:54.262331 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:54.262397 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:54.262547 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:17:54.262777 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:54.264350 (+ 1573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:54.265480 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:54.266047 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:54.266170 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:54.266943 (+ 773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:17:54.266947 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:17:54.266948 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:17:54.267177 (+ 229us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:17:54.267212 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:17:54.267223 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:17:54.267313 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0929 03:17:54.267433 (+ 120us) server_negotiation.cc:299] Negotiation successful
0929 03:17:54.267518 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":81,"threads_started":1}
I20260929 03:17:54.268615 21193 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 03:17:54.279085 26496 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:17:54.279613 26496 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:17:54.287081 26496 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e7404f5acacf4a0a9ba7a6228b78ee2d: Generated new TSK 0
I20260929 03:17:54.287285 26496 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260929 03:17:54.315137 26514 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260929 03:17:54.315219 26514 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260929 03:17:54.315485 26488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:54.293233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50074 (local address 127.20.178.126:43457)
0929 03:17:54.293280 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:54.293282 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:54.293367 (+ 85us) server_negotiation.cc:407] Connection header received
0929 03:17:54.293459 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:54.293460 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:54.293472 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:54.293618 (+ 146us) server_negotiation.cc:226] Negotiated authn=JWT
0929 03:17:54.293762 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:54.294263 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:54.295063 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:54.295256 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:54.295341 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:54.295540 (+ 199us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 03:17:54.315276 (+ 19736us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43559/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0929 03:17:54.315426 (+ 150us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:50074: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43559/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":25}
W20260929 03:17:54.315555 26488 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:50074: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43559/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260929 03:17:54.315579 26513 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:17:54.293130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:43457 (local address 127.0.0.1:50074)
0929 03:17:54.293285 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0929 03:17:54.293296 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 03:17:54.293386 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:17:54.293619 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:17:54.293620 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:17:54.293621 (+ 1us) client_negotiation.cc:189] Negotiated authn=JWT
0929 03:17:54.293684 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:54.293685 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:54.294346 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:54.294348 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:54.294947 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:54.294948 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:54.295347 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:54.295348 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:54.295406 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:54.295414 (+ 8us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 03:17:54.315376 (+ 19962us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43559/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0929 03:17:54.315483 (+ 107us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.20.178.126:43457: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43559/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":88,"threads_started":1}
I20260929 03:17:54.315836 21193 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuoaHbX/build/release/bin/kudu with pid 26417
2026-09-29T03:17:54Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (225 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260929 03:17:54.353008 21193 mini_oidc.cc:152] Starting JWKS server
I20260929 03:17:54.353699 21193 webserver.cc:460] Webserver started at https://127.0.0.1:35483/ using document root <none> and password file <none>
I20260929 03:17:54.353778 21193 mini_oidc.cc:189] Starting OIDC Discovery server
I20260929 03:17:54.353940 21193 webserver.cc:460] Webserver started at http://127.0.0.1:34499/ using document root <none> and password file <none>
2026-09-29T03:17:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:17:54Z Disabled control of system clock
I20260929 03:17:54.361588 21193 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuoaHbX/build/release/bin/kudu
/tmp/dist-test-taskuoaHbX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790651852476208-21193-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790651852476208-21193-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790651852476208-21193-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790651852476208-21193-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.178.126:42269
--webserver_interface=127.20.178.126
--webserver_port=0
--builtin_ntp_servers=127.20.178.84:32905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.178.126:42269
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35483/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260929 03:17:54.440057 26520 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:17:54.440244 26520 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:17:54.440274 26520 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:17:54.441370 26520 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:17:54.441406 26520 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:17:54.441418 26520 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:17:54.441431 26520 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:17:54.441442 26520 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 03:17:54.441455 26520 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 03:17:54.441466 26520 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:35483/jwks.json
I20260929 03:17:54.442747 26520 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.178.84:32905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790651852476208-21193-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790651852476208-21193-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.178.126:42269
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.178.126:42269
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35483/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790651852476208-21193-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.178.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26520
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790651852476208-21193-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:17:54.442942 26520 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:17:54.443176 26520 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:17:54.447984 26526 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:17:54.448037 26528 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:17:54.448105 26520 server_base.cc:1034] running on GCE node
W20260929 03:17:54.448073 26525 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:17:54.448534 26520 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:17:54.448874 26520 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:17:54.450039 26520 hybrid_clock.cc:648] HybridClock initialized: now 1790651874450017 us; error 38 us; skew 500 ppm
I20260929 03:17:54.451267 26520 webserver.cc:460] Webserver started at http://127.20.178.126:40701/ using document root <none> and password file <none>
I20260929 03:17:54.451519 26520 fs_manager.cc:359] Metadata directory not provided
I20260929 03:17:54.451588 26520 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:17:54.451720 26520 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:17:54.452836 26520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790651852476208-21193-0/minicluster-data/master-0/data/instance:
uuid: "6630679e92a04365b44810f2e5442486"
format_stamp: "Formatted at 2026-09-29 03:17:54 on dist-test-slave-c9qh"
server_key: "141400d5a20dc028e44c51689237b866"
server_key_iv: "99b8ed224c870e441dc3c1333777999e"
server_key_version: "encryptionkey@0"
I20260929 03:17:54.453176 26520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790651852476208-21193-0/minicluster-data/master-0/wal/instance:
uuid: "6630679e92a04365b44810f2e5442486"
format_stamp: "Formatted at 2026-09-29 03:17:54 on dist-test-slave-c9qh"
server_key: "141400d5a20dc028e44c51689237b866"
server_key_iv: "99b8ed224c870e441dc3c1333777999e"
server_key_version: "encryptionkey@0"
I20260929 03:17:54.454461 26520 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 03:17:54.455238 26534 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:17:54.455412 26520 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 03:17:54.455510 26520 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790651852476208-21193-0/minicluster-data/master-0/data,/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790651852476208-21193-0/minicluster-data/master-0/wal
uuid: "6630679e92a04365b44810f2e5442486"
format_stamp: "Formatted at 2026-09-29 03:17:54 on dist-test-slave-c9qh"
server_key: "141400d5a20dc028e44c51689237b866"
server_key_iv: "99b8ed224c870e441dc3c1333777999e"
server_key_version: "encryptionkey@0"
I20260929 03:17:54.455590 26520 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790651852476208-21193-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790651852476208-21193-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790651852476208-21193-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:17:54.464087 26520 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:17:54.464353 26520 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:17:54.464501 26520 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:17:54.468263 26520 rpc_server.cc:307] RPC server started. Bound to: 127.20.178.126:42269
I20260929 03:17:54.468290 26586 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.178.126:42269 every 8 connection(s)
I20260929 03:17:54.468602 26520 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790651852476208-21193-0/minicluster-data/master-0/data/info.pb
I20260929 03:17:54.469280 26587 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:17:54.471977 26587 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6630679e92a04365b44810f2e5442486: Bootstrap starting.
I20260929 03:17:54.472577 26587 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6630679e92a04365b44810f2e5442486: Neither blocks nor log segments found. Creating new log.
I20260929 03:17:54.472905 26587 log.cc:824] T 00000000000000000000000000000000 P 6630679e92a04365b44810f2e5442486: Log is configured to *not* fsync() on all Append() calls
I20260929 03:17:54.473605 26587 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6630679e92a04365b44810f2e5442486: No bootstrap required, opened a new log
I20260929 03:17:54.475101 26587 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6630679e92a04365b44810f2e5442486 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6630679e92a04365b44810f2e5442486" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 42269 } }
I20260929 03:17:54.475229 26587 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6630679e92a04365b44810f2e5442486 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:17:54.475274 26587 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6630679e92a04365b44810f2e5442486 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6630679e92a04365b44810f2e5442486, State: Initialized, Role: FOLLOWER
I20260929 03:17:54.475449 26587 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6630679e92a04365b44810f2e5442486 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6630679e92a04365b44810f2e5442486" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 42269 } }
I20260929 03:17:54.475510 26587 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6630679e92a04365b44810f2e5442486 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:17:54.475523 21193 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuoaHbX/build/release/bin/kudu as pid 26520
I20260929 03:17:54.475569 26587 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6630679e92a04365b44810f2e5442486 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:17:54.475620 21193 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790651852476208-21193-0/minicluster-data/master-0/wal/instance
I20260929 03:17:54.475617 26587 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6630679e92a04365b44810f2e5442486 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:17:54.475776 21193 external_mini_cluster.cc:1428] Setting key 3e3e2aff8827ea02ce667b42b81d924c
I20260929 03:17:54.476282 26587 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6630679e92a04365b44810f2e5442486 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6630679e92a04365b44810f2e5442486" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 42269 } }
I20260929 03:17:54.476399 26587 leader_election.cc:302] T 00000000000000000000000000000000 P 6630679e92a04365b44810f2e5442486 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6630679e92a04365b44810f2e5442486; no voters:
I20260929 03:17:54.476562 26587 leader_election.cc:288] T 00000000000000000000000000000000 P 6630679e92a04365b44810f2e5442486 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:17:54.476583 26592 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6630679e92a04365b44810f2e5442486 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:17:54.476756 26592 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6630679e92a04365b44810f2e5442486 [term 1 LEADER]: Becoming Leader. State: Replica: 6630679e92a04365b44810f2e5442486, State: Running, Role: LEADER
I20260929 03:17:54.476840 26587 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6630679e92a04365b44810f2e5442486 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:17:54.476836 26592 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6630679e92a04365b44810f2e5442486 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6630679e92a04365b44810f2e5442486" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 42269 } }
I20260929 03:17:54.477161 26592 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6630679e92a04365b44810f2e5442486 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6630679e92a04365b44810f2e5442486. Latest consensus state: current_term: 1 leader_uuid: "6630679e92a04365b44810f2e5442486" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6630679e92a04365b44810f2e5442486" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 42269 } } }
I20260929 03:17:54.477236 26592 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6630679e92a04365b44810f2e5442486 [sys.catalog]: This master's current role is: LEADER
I20260929 03:17:54.477196 26593 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6630679e92a04365b44810f2e5442486 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6630679e92a04365b44810f2e5442486" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6630679e92a04365b44810f2e5442486" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 42269 } } }
I20260929 03:17:54.477428 26593 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6630679e92a04365b44810f2e5442486 [sys.catalog]: This master's current role is: LEADER
I20260929 03:17:54.477478 26597 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:17:54.478096 26597 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:17:54.481272 26597 catalog_manager.cc:1347] Generated new cluster ID: 82a386a3cb1b4c7188730fbeb570a043
I20260929 03:17:54.481331 26597 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:17:54.481916 26591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:54.476264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51422 (local address 127.20.178.126:42269)
0929 03:17:54.476549 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:54.476558 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:54.476589 (+ 31us) server_negotiation.cc:407] Connection header received
0929 03:17:54.476721 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:54.476728 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:54.476803 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:54.477200 (+ 397us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:17:54.477320 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:54.478830 (+ 1510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:54.479740 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:54.480156 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:54.480298 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:54.481188 (+ 890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:17:54.481192 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:17:54.481192 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:17:54.481400 (+ 208us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:17:54.481438 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:17:54.481451 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:17:54.481507 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0929 03:17:54.481630 (+ 123us) server_negotiation.cc:299] Negotiation successful
0929 03:17:54.481719 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":191,"thread_start_us":98,"threads_started":1}
I20260929 03:17:54.482847 21193 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 03:17:54.502828 26597 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:17:54.503525 26597 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:17:54.507781 26597 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6630679e92a04365b44810f2e5442486: Generated new TSK 0
I20260929 03:17:54.507962 26597 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 03:17:54.525671 26591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:54.522212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51424 (local address 127.20.178.126:42269)
0929 03:17:54.522278 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:54.522280 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:54.522378 (+ 98us) server_negotiation.cc:407] Connection header received
0929 03:17:54.522466 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:54.522466 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:54.522479 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:54.522576 (+ 97us) server_negotiation.cc:226] Negotiated authn=JWT
0929 03:17:54.522716 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:54.523213 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:54.523997 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:54.524217 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:54.524304 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:54.524495 (+ 191us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 03:17:54.525450 (+ 955us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35483/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0929 03:17:54.525583 (+ 133us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:51424: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35483/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":41}
W20260929 03:17:54.525727 26591 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:51424: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35483/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260929 03:17:54.525765 26616 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:17:54.522151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:42269 (local address 127.0.0.1:51424)
0929 03:17:54.522309 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0929 03:17:54.522318 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 03:17:54.522396 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:17:54.522576 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:17:54.522577 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:17:54.522579 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0929 03:17:54.522635 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:54.522637 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:54.523318 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:54.523319 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:54.523912 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:54.523914 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:54.524312 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:54.524314 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:54.524372 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:54.524379 (+ 7us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 03:17:54.525552 (+ 1173us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35483/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0929 03:17:54.525655 (+ 103us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.20.178.126:42269: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35483/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":97,"threads_started":1}
I20260929 03:17:54.526382 21193 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuoaHbX/build/release/bin/kudu with pid 26520
E20260929 03:17:54.537076 26617 webserver.cc:533] Webserver: sslize failed:
E20260929 03:17:54.537154 26617 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-29T03:17:54Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (210 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-29T03:17:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:17:54Z Disabled control of system clock
I20260929 03:17:54.570399 21193 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuoaHbX/build/release/bin/kudu
/tmp/dist-test-taskuoaHbX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790651852476208-21193-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790651852476208-21193-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790651852476208-21193-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790651852476208-21193-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.178.126:36435
--webserver_interface=127.20.178.126
--webserver_port=0
--builtin_ntp_servers=127.20.178.84:43867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.178.126:36435
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790651852476208-21193-0/testchainca.cert with env {}
W20260929 03:17:54.649210 26621 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:17:54.649396 26621 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:17:54.649417 26621 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:17:54.650473 26621 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:17:54.650504 26621 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:17:54.650517 26621 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:17:54.650529 26621 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:17:54.650542 26621 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 03:17:54.650553 26621 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 03:17:54.650565 26621 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260929 03:17:54.651937 26621 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.178.84:43867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790651852476208-21193-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790651852476208-21193-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.178.126:36435
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.178.126:36435
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790651852476208-21193-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.178.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790651852476208-21193-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26621
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790651852476208-21193-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:17:54.652114 26621 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:17:54.652259 26621 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:17:54.656854 26629 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:17:54.656864 26626 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:17:54.657060 26621 server_base.cc:1034] running on GCE node
W20260929 03:17:54.657065 26627 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:17:54.657364 26621 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:17:54.657652 26621 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:17:54.658818 26621 hybrid_clock.cc:648] HybridClock initialized: now 1790651874658802 us; error 46 us; skew 500 ppm
I20260929 03:17:54.659982 26621 webserver.cc:460] Webserver started at http://127.20.178.126:37669/ using document root <none> and password file <none>
I20260929 03:17:54.660238 26621 fs_manager.cc:359] Metadata directory not provided
I20260929 03:17:54.660317 26621 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:17:54.660468 26621 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:17:54.661547 26621 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790651852476208-21193-0/minicluster-data/master-0/data/instance:
uuid: "804b1cb5ab5943febcee65e72e6ff8cf"
format_stamp: "Formatted at 2026-09-29 03:17:54 on dist-test-slave-c9qh"
server_key: "c1ce0f0b6140661010ae920e05f57cc0"
server_key_iv: "7ca2e0238573f25b193c3b95d76d107d"
server_key_version: "encryptionkey@0"
I20260929 03:17:54.661911 26621 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790651852476208-21193-0/minicluster-data/master-0/wal/instance:
uuid: "804b1cb5ab5943febcee65e72e6ff8cf"
format_stamp: "Formatted at 2026-09-29 03:17:54 on dist-test-slave-c9qh"
server_key: "c1ce0f0b6140661010ae920e05f57cc0"
server_key_iv: "7ca2e0238573f25b193c3b95d76d107d"
server_key_version: "encryptionkey@0"
I20260929 03:17:54.663265 26621 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 03:17:54.663977 26635 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:17:54.664151 26621 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 03:17:54.664240 26621 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790651852476208-21193-0/minicluster-data/master-0/data,/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790651852476208-21193-0/minicluster-data/master-0/wal
uuid: "804b1cb5ab5943febcee65e72e6ff8cf"
format_stamp: "Formatted at 2026-09-29 03:17:54 on dist-test-slave-c9qh"
server_key: "c1ce0f0b6140661010ae920e05f57cc0"
server_key_iv: "7ca2e0238573f25b193c3b95d76d107d"
server_key_version: "encryptionkey@0"
I20260929 03:17:54.664333 26621 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790651852476208-21193-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790651852476208-21193-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790651852476208-21193-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:17:54.677436 26621 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:17:54.677711 26621 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:17:54.677862 26621 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:17:54.681603 26621 rpc_server.cc:307] RPC server started. Bound to: 127.20.178.126:36435
I20260929 03:17:54.681658 26687 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.178.126:36435 every 8 connection(s)
I20260929 03:17:54.681988 26621 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790651852476208-21193-0/minicluster-data/master-0/data/info.pb
I20260929 03:17:54.682934 26688 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:17:54.684240 21193 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuoaHbX/build/release/bin/kudu as pid 26621
I20260929 03:17:54.684351 21193 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790651852476208-21193-0/minicluster-data/master-0/wal/instance
I20260929 03:17:54.684516 21193 external_mini_cluster.cc:1428] Setting key ebe425214b6a4c3a3a84b8242fdf56ea
I20260929 03:17:54.685492 26688 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 804b1cb5ab5943febcee65e72e6ff8cf: Bootstrap starting.
I20260929 03:17:54.686182 26688 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 804b1cb5ab5943febcee65e72e6ff8cf: Neither blocks nor log segments found. Creating new log.
I20260929 03:17:54.686502 26688 log.cc:824] T 00000000000000000000000000000000 P 804b1cb5ab5943febcee65e72e6ff8cf: Log is configured to *not* fsync() on all Append() calls
I20260929 03:17:54.687211 26688 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 804b1cb5ab5943febcee65e72e6ff8cf: No bootstrap required, opened a new log
I20260929 03:17:54.689415 26688 raft_consensus.cc:348] T 00000000000000000000000000000000 P 804b1cb5ab5943febcee65e72e6ff8cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "804b1cb5ab5943febcee65e72e6ff8cf" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 36435 } }
I20260929 03:17:54.689589 26688 raft_consensus.cc:374] T 00000000000000000000000000000000 P 804b1cb5ab5943febcee65e72e6ff8cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:17:54.689635 26688 raft_consensus.cc:729] T 00000000000000000000000000000000 P 804b1cb5ab5943febcee65e72e6ff8cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 804b1cb5ab5943febcee65e72e6ff8cf, State: Initialized, Role: FOLLOWER
I20260929 03:17:54.689781 26688 consensus_queue.cc:260] T 00000000000000000000000000000000 P 804b1cb5ab5943febcee65e72e6ff8cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "804b1cb5ab5943febcee65e72e6ff8cf" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 36435 } }
I20260929 03:17:54.689893 26688 raft_consensus.cc:388] T 00000000000000000000000000000000 P 804b1cb5ab5943febcee65e72e6ff8cf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:17:54.689931 26688 raft_consensus.cc:482] T 00000000000000000000000000000000 P 804b1cb5ab5943febcee65e72e6ff8cf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:17:54.689980 26688 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 804b1cb5ab5943febcee65e72e6ff8cf [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:17:54.690487 26691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:54.684999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55758 (local address 127.20.178.126:36435)
0929 03:17:54.685218 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:54.685227 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:54.685259 (+ 32us) server_negotiation.cc:407] Connection header received
0929 03:17:54.685770 (+ 511us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:54.685780 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:54.685869 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:54.686015 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:17:54.686191 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:54.687516 (+ 1325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:54.688271 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:54.688957 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:54.689099 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:54.689747 (+ 648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:17:54.689751 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:17:54.689751 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:17:54.689954 (+ 203us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:17:54.689996 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:17:54.690011 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:17:54.690077 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0929 03:17:54.690196 (+ 119us) server_negotiation.cc:299] Negotiation successful
0929 03:17:54.690293 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":89,"threads_started":1}
I20260929 03:17:54.690565 26688 raft_consensus.cc:504] T 00000000000000000000000000000000 P 804b1cb5ab5943febcee65e72e6ff8cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "804b1cb5ab5943febcee65e72e6ff8cf" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 36435 } }
I20260929 03:17:54.690685 26688 leader_election.cc:302] T 00000000000000000000000000000000 P 804b1cb5ab5943febcee65e72e6ff8cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 804b1cb5ab5943febcee65e72e6ff8cf; no voters:
I20260929 03:17:54.690836 26688 leader_election.cc:288] T 00000000000000000000000000000000 P 804b1cb5ab5943febcee65e72e6ff8cf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:17:54.690892 26693 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 804b1cb5ab5943febcee65e72e6ff8cf [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:17:54.691008 26693 raft_consensus.cc:686] T 00000000000000000000000000000000 P 804b1cb5ab5943febcee65e72e6ff8cf [term 1 LEADER]: Becoming Leader. State: Replica: 804b1cb5ab5943febcee65e72e6ff8cf, State: Running, Role: LEADER
I20260929 03:17:54.691119 26688 sys_catalog.cc:564] T 00000000000000000000000000000000 P 804b1cb5ab5943febcee65e72e6ff8cf [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:17:54.691095 26693 consensus_queue.cc:237] T 00000000000000000000000000000000 P 804b1cb5ab5943febcee65e72e6ff8cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "804b1cb5ab5943febcee65e72e6ff8cf" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 36435 } }
I20260929 03:17:54.692680 21193 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 03:17:54.692950 26695 sys_catalog.cc:455] T 00000000000000000000000000000000 P 804b1cb5ab5943febcee65e72e6ff8cf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "804b1cb5ab5943febcee65e72e6ff8cf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "804b1cb5ab5943febcee65e72e6ff8cf" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 36435 } } }
I20260929 03:17:54.693200 26695 sys_catalog.cc:458] T 00000000000000000000000000000000 P 804b1cb5ab5943febcee65e72e6ff8cf [sys.catalog]: This master's current role is: LEADER
I20260929 03:17:54.693329 26694 sys_catalog.cc:455] T 00000000000000000000000000000000 P 804b1cb5ab5943febcee65e72e6ff8cf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 804b1cb5ab5943febcee65e72e6ff8cf. Latest consensus state: current_term: 1 leader_uuid: "804b1cb5ab5943febcee65e72e6ff8cf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "804b1cb5ab5943febcee65e72e6ff8cf" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 36435 } } }
I20260929 03:17:54.693369 26694 sys_catalog.cc:458] T 00000000000000000000000000000000 P 804b1cb5ab5943febcee65e72e6ff8cf [sys.catalog]: This master's current role is: LEADER
W20260929 03:17:54.693917 26707 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 804b1cb5ab5943febcee65e72e6ff8cf: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 03:17:54.693974 26707 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 03:17:54.694024 26708 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:17:54.694172 26708 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:17:54.695564 26708 catalog_manager.cc:1347] Generated new cluster ID: 67bcccebdbbb401297723b2fe20fe758
I20260929 03:17:54.695624 26708 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:17:54.703289 26708 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:17:54.703791 26708 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:17:54.707425 26708 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 804b1cb5ab5943febcee65e72e6ff8cf: Generated new TSK 0
I20260929 03:17:54.707608 26708 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 03:17:55.143982 26691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:54.713471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55774 (local address 127.20.178.126:36435)
0929 03:17:54.713537 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:54.713539 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:54.713664 (+ 125us) server_negotiation.cc:407] Connection header received
0929 03:17:54.713735 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:54.713735 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:54.713745 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:54.713811 (+ 66us) server_negotiation.cc:226] Negotiated authn=JWT
0929 03:17:54.713959 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:54.714423 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:54.715558 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:54.715752 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:54.715810 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:54.715999 (+ 189us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 03:17:55.143764 (+427765us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0929 03:17:55.143922 (+ 158us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:55774: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":41}
W20260929 03:17:55.144080 26691 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:55774: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260929 03:17:55.144102 26717 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:17:54.713419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:36435 (local address 127.0.0.1:55774)
0929 03:17:54.713586 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0929 03:17:54.713598 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 03:17:54.713669 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:17:54.713833 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:17:54.713834 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:17:54.713836 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0929 03:17:54.713898 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:54.713899 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:54.714510 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:54.714511 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:54.715472 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:54.715475 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:54.715839 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:54.715841 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:54.715901 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:54.715907 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 03:17:55.143903 (+427996us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0929 03:17:55.144023 (+ 120us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.20.178.126:36435: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":98,"threads_started":1}
I20260929 03:17:55.144558 21193 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuoaHbX/build/release/bin/kudu with pid 26621
2026-09-29T03:17:55Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (599 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (86 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (87 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651852476208-21193-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:17:55 dist-test-slave-c9qh krb5kdc[26729](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:17:55 dist-test-slave-c9qh krb5kdc[26729](info): set up 2 sockets
Sep 29 03:17:55 dist-test-slave-c9qh krb5kdc[26729](info): commencing operation
krb5kdc: starting...
W20260929 03:17:56.356878 21193 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651852476208-21193-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651852476208-21193-0/krb5kdc/test-user.keytab.
Sep 29 03:17:56 dist-test-slave-c9qh krb5kdc[26729](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651876, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:17:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:17:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.178.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.178.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.178.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.126.keytab.
Entry for principal kudu/127.20.178.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.126.keytab.
I20260929 03:17:56.491451 21193 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuoaHbX/build/release/bin/kudu
/tmp/dist-test-taskuoaHbX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651852476208-21193-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651852476208-21193-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651852476208-21193-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651852476208-21193-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.178.126:43487
--webserver_interface=127.20.178.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.126.keytab
--principal=kudu/127.20.178.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.178.84:35359
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.178.126:43487
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651852476208-21193-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651852476208-21193-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651852476208-21193-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:17:56.572304 26742 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:17:56.572496 26742 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:17:56.572517 26742 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:17:56.573845 26742 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:17:56.573887 26742 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:17:56.573901 26742 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:17:56.573915 26742 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:17:56.573966 26742 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:17:56.575482 26742 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.178.84:35359
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651852476208-21193-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651852476208-21193-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.178.126:43487
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.126.keytab
--principal=kudu/127.20.178.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.178.126:43487
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651852476208-21193-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.178.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26742
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651852476208-21193-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:17:56.575727 26742 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:17:56.575940 26742 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:17:56.580700 26748 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:17:56.580708 26750 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:17:56.580717 26747 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:17:56.580865 26742 server_base.cc:1034] running on GCE node
I20260929 03:17:56.581090 26742 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:17:56.581374 26742 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:17:56.582538 26742 hybrid_clock.cc:648] HybridClock initialized: now 1790651876582524 us; error 25 us; skew 500 ppm
Sep 29 03:17:56 dist-test-slave-c9qh krb5kdc[26729](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651876, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:17:56.584919 26742 init.cc:377] Logged in from keytab as kudu/127.20.178.126@KRBTEST.COM (short username kudu)
I20260929 03:17:56.585369 26742 webserver.cc:460] Webserver started at http://127.20.178.126:37997/ using document root <none> and password file <none>
I20260929 03:17:56.585652 26742 fs_manager.cc:359] Metadata directory not provided
I20260929 03:17:56.585727 26742 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:17:56.585865 26742 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:17:56.586892 26742 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651852476208-21193-0/minicluster-data/master-0/data/instance:
uuid: "96b54299401e401b9d1b0d059d2cff58"
format_stamp: "Formatted at 2026-09-29 03:17:56 on dist-test-slave-c9qh"
server_key: "56aea140a3636587a811809061a497d7"
server_key_iv: "6ae25dba9e76d8b25ffe21ba35619c90"
server_key_version: "encryptionkey@0"
I20260929 03:17:56.587265 26742 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651852476208-21193-0/minicluster-data/master-0/wal/instance:
uuid: "96b54299401e401b9d1b0d059d2cff58"
format_stamp: "Formatted at 2026-09-29 03:17:56 on dist-test-slave-c9qh"
server_key: "56aea140a3636587a811809061a497d7"
server_key_iv: "6ae25dba9e76d8b25ffe21ba35619c90"
server_key_version: "encryptionkey@0"
I20260929 03:17:56.588562 26742 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260929 03:17:56.589325 26757 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:17:56.589490 26742 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 03:17:56.589592 26742 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651852476208-21193-0/minicluster-data/master-0/data,/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651852476208-21193-0/minicluster-data/master-0/wal
uuid: "96b54299401e401b9d1b0d059d2cff58"
format_stamp: "Formatted at 2026-09-29 03:17:56 on dist-test-slave-c9qh"
server_key: "56aea140a3636587a811809061a497d7"
server_key_iv: "6ae25dba9e76d8b25ffe21ba35619c90"
server_key_version: "encryptionkey@0"
I20260929 03:17:56.589663 26742 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651852476208-21193-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651852476208-21193-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651852476208-21193-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:17:56.610071 26742 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:17:56.612440 26742 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:17:56.612617 26742 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:17:56.616580 26742 rpc_server.cc:307] RPC server started. Bound to: 127.20.178.126:43487
I20260929 03:17:56.616616 26809 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.178.126:43487 every 8 connection(s)
I20260929 03:17:56.617005 26742 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651852476208-21193-0/minicluster-data/master-0/data/info.pb
I20260929 03:17:56.617849 26810 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:17:56.620110 26810 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 96b54299401e401b9d1b0d059d2cff58: Bootstrap starting.
I20260929 03:17:56.620736 26810 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 96b54299401e401b9d1b0d059d2cff58: Neither blocks nor log segments found. Creating new log.
I20260929 03:17:56.621048 26810 log.cc:824] T 00000000000000000000000000000000 P 96b54299401e401b9d1b0d059d2cff58: Log is configured to *not* fsync() on all Append() calls
I20260929 03:17:56.621737 26810 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 96b54299401e401b9d1b0d059d2cff58: No bootstrap required, opened a new log
I20260929 03:17:56.623333 26810 raft_consensus.cc:348] T 00000000000000000000000000000000 P 96b54299401e401b9d1b0d059d2cff58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96b54299401e401b9d1b0d059d2cff58" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 43487 } }
I20260929 03:17:56.623468 26810 raft_consensus.cc:374] T 00000000000000000000000000000000 P 96b54299401e401b9d1b0d059d2cff58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:17:56.623492 26810 raft_consensus.cc:729] T 00000000000000000000000000000000 P 96b54299401e401b9d1b0d059d2cff58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96b54299401e401b9d1b0d059d2cff58, State: Initialized, Role: FOLLOWER
I20260929 03:17:56.623643 26810 consensus_queue.cc:260] T 00000000000000000000000000000000 P 96b54299401e401b9d1b0d059d2cff58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96b54299401e401b9d1b0d059d2cff58" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 43487 } }
I20260929 03:17:56.623705 26810 raft_consensus.cc:388] T 00000000000000000000000000000000 P 96b54299401e401b9d1b0d059d2cff58 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:17:56.623745 26810 raft_consensus.cc:482] T 00000000000000000000000000000000 P 96b54299401e401b9d1b0d059d2cff58 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:17:56.623797 26810 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 96b54299401e401b9d1b0d059d2cff58 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:17:56.624327 26810 raft_consensus.cc:504] T 00000000000000000000000000000000 P 96b54299401e401b9d1b0d059d2cff58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96b54299401e401b9d1b0d059d2cff58" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 43487 } }
I20260929 03:17:56.624408 26810 leader_election.cc:302] T 00000000000000000000000000000000 P 96b54299401e401b9d1b0d059d2cff58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 96b54299401e401b9d1b0d059d2cff58; no voters:
I20260929 03:17:56.624614 26810 leader_election.cc:288] T 00000000000000000000000000000000 P 96b54299401e401b9d1b0d059d2cff58 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:17:56.624703 26813 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 96b54299401e401b9d1b0d059d2cff58 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:17:56.624908 26810 sys_catalog.cc:564] T 00000000000000000000000000000000 P 96b54299401e401b9d1b0d059d2cff58 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:17:56.625093 26813 raft_consensus.cc:686] T 00000000000000000000000000000000 P 96b54299401e401b9d1b0d059d2cff58 [term 1 LEADER]: Becoming Leader. State: Replica: 96b54299401e401b9d1b0d059d2cff58, State: Running, Role: LEADER
I20260929 03:17:56.625216 26813 consensus_queue.cc:237] T 00000000000000000000000000000000 P 96b54299401e401b9d1b0d059d2cff58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96b54299401e401b9d1b0d059d2cff58" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 43487 } }
I20260929 03:17:56.625586 26815 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96b54299401e401b9d1b0d059d2cff58 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 96b54299401e401b9d1b0d059d2cff58. Latest consensus state: current_term: 1 leader_uuid: "96b54299401e401b9d1b0d059d2cff58" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96b54299401e401b9d1b0d059d2cff58" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 43487 } } }
I20260929 03:17:56.625586 26814 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96b54299401e401b9d1b0d059d2cff58 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "96b54299401e401b9d1b0d059d2cff58" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96b54299401e401b9d1b0d059d2cff58" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 43487 } } }
I20260929 03:17:56.625701 26815 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96b54299401e401b9d1b0d059d2cff58 [sys.catalog]: This master's current role is: LEADER
I20260929 03:17:56.625707 26814 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96b54299401e401b9d1b0d059d2cff58 [sys.catalog]: This master's current role is: LEADER
I20260929 03:17:56.626104 21193 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuoaHbX/build/release/bin/kudu as pid 26742
I20260929 03:17:56.626228 21193 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651852476208-21193-0/minicluster-data/master-0/wal/instance
I20260929 03:17:56.626387 21193 external_mini_cluster.cc:1428] Setting key 7c848b6a89494fad823baaba4b8ebdfd
I20260929 03:17:56.626634 26822 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:17:56.627164 26822 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:17:56.629207 26822 catalog_manager.cc:1347] Generated new cluster ID: 1b726d1d1e444d1c971791c798f89c98
I20260929 03:17:56.629261 26822 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 29 03:17:56 dist-test-slave-c9qh krb5kdc[26729](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651876, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.178.126@KRBTEST.COM
I20260929 03:17:56.637977 26831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:56.627036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51888 (local address 127.20.178.126:43487)
0929 03:17:56.627222 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:56.627232 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:56.627267 (+ 35us) server_negotiation.cc:407] Connection header received
0929 03:17:56.627394 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:56.627472 (+ 78us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:56.627531 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:56.627691 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:17:56.628333 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:56.629718 (+ 1385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:56.630727 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:56.631361 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:56.631479 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:56.634346 (+ 2867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:17:56.634350 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:17:56.634350 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:17:56.634362 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:17:56.636267 (+ 1905us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:56.636673 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:56.636676 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:56.636676 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:56.636775 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:56.637011 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:56.637011 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:56.637012 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:56.637312 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:17:56.637441 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0929 03:17:56.637655 (+ 214us) server_negotiation.cc:299] Negotiation successful
0929 03:17:56.637779 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":75,"threads_started":1}
I20260929 03:17:56.641094 26822 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:17:56.641780 26822 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:17:56.645574 26822 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 96b54299401e401b9d1b0d059d2cff58: Generated new TSK 0
I20260929 03:17:56.645807 26822 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.178.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.178.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.178.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.65.keytab.
Entry for principal kudu/127.20.178.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.65.keytab.
I20260929 03:17:56.675734 21193 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuoaHbX/build/release/bin/kudu
/tmp/dist-test-taskuoaHbX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651852476208-21193-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651852476208-21193-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651852476208-21193-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651852476208-21193-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.178.65:0
--local_ip_for_outbound_sockets=127.20.178.65
--webserver_interface=127.20.178.65
--webserver_port=0
--tserver_master_addrs=127.20.178.126:43487
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.65.keytab
--principal=kudu/127.20.178.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.178.84:35359
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651852476208-21193-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651852476208-21193-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651852476208-21193-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:17:56.755079 26836 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:17:56.755280 26836 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:17:56.755303 26836 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:17:56.756359 26836 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:17:56.756392 26836 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:17:56.756417 26836 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.178.65
I20260929 03:17:56.757783 26836 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.178.84:35359
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651852476208-21193-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651852476208-21193-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.65.keytab
--principal=kudu/127.20.178.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.178.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651852476208-21193-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.178.65
--webserver_port=0
--tserver_master_addrs=127.20.178.126:43487
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26836
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.178.65
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651852476208-21193-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:17:56.757953 26836 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:17:56.758111 26836 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:17:56.762969 26841 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:17:56.762929 26844 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:17:56.762933 26842 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:17:56.763181 26836 server_base.cc:1034] running on GCE node
I20260929 03:17:56.763427 26836 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:17:56.763664 26836 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:17:56.764815 26836 hybrid_clock.cc:648] HybridClock initialized: now 1790651876764806 us; error 24 us; skew 500 ppm
Sep 29 03:17:56 dist-test-slave-c9qh krb5kdc[26729](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651876, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:17:56.767333 26836 init.cc:377] Logged in from keytab as kudu/127.20.178.65@KRBTEST.COM (short username kudu)
I20260929 03:17:56.767704 26836 webserver.cc:460] Webserver started at http://127.20.178.65:44471/ using document root <none> and password file <none>
I20260929 03:17:56.767875 26836 fs_manager.cc:359] Metadata directory not provided
I20260929 03:17:56.767913 26836 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:17:56.767997 26836 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:17:56.769023 26836 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651852476208-21193-0/minicluster-data/ts-0/data/instance:
uuid: "a0e89f6091754d6c9496324692d4c973"
format_stamp: "Formatted at 2026-09-29 03:17:56 on dist-test-slave-c9qh"
server_key: "4658d80174f0af403be581b44d31b662"
server_key_iv: "daac1757f3a525ad076d42935c06118b"
server_key_version: "encryptionkey@0"
I20260929 03:17:56.769320 26836 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651852476208-21193-0/minicluster-data/ts-0/wal/instance:
uuid: "a0e89f6091754d6c9496324692d4c973"
format_stamp: "Formatted at 2026-09-29 03:17:56 on dist-test-slave-c9qh"
server_key: "4658d80174f0af403be581b44d31b662"
server_key_iv: "daac1757f3a525ad076d42935c06118b"
server_key_version: "encryptionkey@0"
I20260929 03:17:56.770618 26836 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 03:17:56.771309 26851 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:17:56.771523 26836 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 03:17:56.771665 26836 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651852476208-21193-0/minicluster-data/ts-0/data,/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651852476208-21193-0/minicluster-data/ts-0/wal
uuid: "a0e89f6091754d6c9496324692d4c973"
format_stamp: "Formatted at 2026-09-29 03:17:56 on dist-test-slave-c9qh"
server_key: "4658d80174f0af403be581b44d31b662"
server_key_iv: "daac1757f3a525ad076d42935c06118b"
server_key_version: "encryptionkey@0"
I20260929 03:17:56.771723 26836 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651852476208-21193-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651852476208-21193-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651852476208-21193-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:17:56.781133 26836 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:17:56.783465 26836 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:17:56.783587 26836 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:17:56.783757 26836 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:17:56.783994 26836 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:17:56.784025 26836 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:17:56.784049 26836 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:17:56.784065 26836 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:17:56.790163 26836 rpc_server.cc:307] RPC server started. Bound to: 127.20.178.65:33489
I20260929 03:17:56.790244 26964 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.178.65:33489 every 8 connection(s)
I20260929 03:17:56.790575 26836 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651852476208-21193-0/minicluster-data/ts-0/data/info.pb
Sep 29 03:17:56 dist-test-slave-c9qh krb5kdc[26729](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651876, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.65@KRBTEST.COM for kudu/127.20.178.126@KRBTEST.COM
I20260929 03:17:56.800078 21193 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuoaHbX/build/release/bin/kudu as pid 26836
I20260929 03:17:56.800105 26831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:56.791309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.65:50163 (local address 127.20.178.126:43487)
0929 03:17:56.791408 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:56.791410 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:56.791601 (+ 191us) server_negotiation.cc:407] Connection header received
0929 03:17:56.792692 (+ 1091us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:56.792693 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:56.792706 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:56.792779 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:17:56.793384 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:56.793903 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:56.794899 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:56.795132 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:56.795218 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:56.797335 (+ 2117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:17:56.797337 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:17:56.797337 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:17:56.797346 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:17:56.798914 (+ 1568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:56.799253 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:56.799254 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:56.799254 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:56.799305 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:56.799506 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:56.799506 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:56.799507 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:56.799654 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:17:56.799716 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0929 03:17:56.799965 (+ 249us) server_negotiation.cc:299] Negotiation successful
0929 03:17:56.800036 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 03:17:56.800197 21193 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651852476208-21193-0/minicluster-data/ts-0/wal/instance
I20260929 03:17:56.800364 26967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:56.791275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:43487 (local address 127.20.178.65:50163)
0929 03:17:56.791459 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0929 03:17:56.791507 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0929 03:17:56.792468 (+ 961us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:17:56.792852 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:17:56.792856 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:17:56.793110 (+ 254us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:17:56.793286 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:56.793290 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:56.794000 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:56.794001 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:56.794792 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:56.794795 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:56.795229 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:56.795230 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:56.795353 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:56.795779 (+ 426us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:17:56.795786 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:17:56.797242 (+ 1456us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:17:56.798984 (+ 1742us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:17:56.798989 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:17:56.798990 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:17:56.799188 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:17:56.799363 (+ 175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:17:56.799363 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:17:56.799364 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:17:56.799451 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:17:56.799714 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:17:56.799718 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:17:56.799822 (+ 104us) client_negotiation.cc:769] Sending connection context
0929 03:17:56.799948 (+ 126us) client_negotiation.cc:240] Negotiation successful
0929 03:17:56.800115 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":89,"threads_started":1}
I20260929 03:17:56.800384 21193 external_mini_cluster.cc:1428] Setting key 6c72f22b5eda856a11cfab9e671b9c48
I20260929 03:17:56.800985 26965 heartbeater.cc:347] Connected to a master server at 127.20.178.126:43487
I20260929 03:17:56.801098 26965 heartbeater.cc:464] Registering TS with master...
I20260929 03:17:56.801285 26965 heartbeater.cc:511] Master 127.20.178.126:43487 requested a full tablet report, sending...
I20260929 03:17:56.801905 26774 ts_manager.cc:194] Registered new tserver with Master: a0e89f6091754d6c9496324692d4c973 (127.20.178.65:33489)
I20260929 03:17:56.802737 26774 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.178.65@KRBTEST.COM'} at 127.20.178.65:50163
WARNING: no policy specified for kudu/127.20.178.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.178.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.178.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.66.keytab.
Entry for principal kudu/127.20.178.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.66.keytab.
I20260929 03:17:56.825666 21193 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuoaHbX/build/release/bin/kudu
/tmp/dist-test-taskuoaHbX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651852476208-21193-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651852476208-21193-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651852476208-21193-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651852476208-21193-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.178.66:0
--local_ip_for_outbound_sockets=127.20.178.66
--webserver_interface=127.20.178.66
--webserver_port=0
--tserver_master_addrs=127.20.178.126:43487
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.66.keytab
--principal=kudu/127.20.178.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.178.84:35359
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651852476208-21193-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651852476208-21193-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651852476208-21193-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:17:56.908373 26970 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:17:56.908617 26970 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:17:56.908651 26970 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:17:56.909825 26970 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:17:56.909894 26970 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:17:56.909938 26970 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.178.66
I20260929 03:17:56.911389 26970 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.178.84:35359
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651852476208-21193-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651852476208-21193-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.66.keytab
--principal=kudu/127.20.178.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.178.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651852476208-21193-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.178.66
--webserver_port=0
--tserver_master_addrs=127.20.178.126:43487
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26970
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.178.66
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651852476208-21193-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:17:56.911648 26970 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:17:56.911875 26970 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:17:56.916741 26978 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:17:56.916818 26970 server_base.cc:1034] running on GCE node
W20260929 03:17:56.916822 26976 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:17:56.916742 26975 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:17:56.917201 26970 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:17:56.917392 26970 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:17:56.918527 26970 hybrid_clock.cc:648] HybridClock initialized: now 1790651876918514 us; error 28 us; skew 500 ppm
Sep 29 03:17:56 dist-test-slave-c9qh krb5kdc[26729](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651876, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:17:56.921046 26970 init.cc:377] Logged in from keytab as kudu/127.20.178.66@KRBTEST.COM (short username kudu)
I20260929 03:17:56.921497 26970 webserver.cc:460] Webserver started at http://127.20.178.66:34591/ using document root <none> and password file <none>
I20260929 03:17:56.921725 26970 fs_manager.cc:359] Metadata directory not provided
I20260929 03:17:56.921792 26970 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:17:56.921940 26970 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:17:56.922951 26970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651852476208-21193-0/minicluster-data/ts-1/data/instance:
uuid: "c1317fbe94e6459e8e4e0f0795d5f3a3"
format_stamp: "Formatted at 2026-09-29 03:17:56 on dist-test-slave-c9qh"
server_key: "e57dee0f828f8921eda36b8ebc413cf7"
server_key_iv: "2e56d375c556afc9794d698015cb9a19"
server_key_version: "encryptionkey@0"
I20260929 03:17:56.923318 26970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651852476208-21193-0/minicluster-data/ts-1/wal/instance:
uuid: "c1317fbe94e6459e8e4e0f0795d5f3a3"
format_stamp: "Formatted at 2026-09-29 03:17:56 on dist-test-slave-c9qh"
server_key: "e57dee0f828f8921eda36b8ebc413cf7"
server_key_iv: "2e56d375c556afc9794d698015cb9a19"
server_key_version: "encryptionkey@0"
I20260929 03:17:56.924614 26970 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 03:17:56.925592 26985 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:17:56.925750 26970 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 03:17:56.925853 26970 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651852476208-21193-0/minicluster-data/ts-1/data,/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651852476208-21193-0/minicluster-data/ts-1/wal
uuid: "c1317fbe94e6459e8e4e0f0795d5f3a3"
format_stamp: "Formatted at 2026-09-29 03:17:56 on dist-test-slave-c9qh"
server_key: "e57dee0f828f8921eda36b8ebc413cf7"
server_key_iv: "2e56d375c556afc9794d698015cb9a19"
server_key_version: "encryptionkey@0"
I20260929 03:17:56.925906 26970 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651852476208-21193-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651852476208-21193-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651852476208-21193-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:17:56.943027 26970 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:17:56.945279 26970 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:17:56.945392 26970 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:17:56.945569 26970 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:17:56.945886 26970 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:17:56.945921 26970 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:17:56.945943 26970 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:17:56.945988 26970 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:17:56.951594 26970 rpc_server.cc:307] RPC server started. Bound to: 127.20.178.66:38271
I20260929 03:17:56.951640 27098 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.178.66:38271 every 8 connection(s)
I20260929 03:17:56.951938 26970 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651852476208-21193-0/minicluster-data/ts-1/data/info.pb
Sep 29 03:17:56 dist-test-slave-c9qh krb5kdc[26729](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651876, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.66@KRBTEST.COM for kudu/127.20.178.126@KRBTEST.COM
I20260929 03:17:56.959969 21193 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuoaHbX/build/release/bin/kudu as pid 26970
I20260929 03:17:56.960080 21193 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651852476208-21193-0/minicluster-data/ts-1/wal/instance
I20260929 03:17:56.960290 21193 external_mini_cluster.cc:1428] Setting key cf57c425a8a5a30bc78941a4966b16dd
I20260929 03:17:56.961163 26831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:56.952770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.66:47331 (local address 127.20.178.126:43487)
0929 03:17:56.952823 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:56.952825 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:56.952926 (+ 101us) server_negotiation.cc:407] Connection header received
0929 03:17:56.953836 (+ 910us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:56.953837 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:56.953846 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:56.953896 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:17:56.954448 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:56.954924 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:56.955900 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:56.956118 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:56.956186 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:56.958264 (+ 2078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:17:56.958266 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:17:56.958266 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:17:56.958272 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:17:56.959618 (+ 1346us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:56.960079 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:56.960080 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:56.960080 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:56.960142 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:56.960432 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:56.960433 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:56.960433 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:56.960606 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:17:56.960706 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0929 03:17:56.961006 (+ 300us) server_negotiation.cc:299] Negotiation successful
0929 03:17:56.961101 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260929 03:17:56.961504 27101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:56.952593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:43487 (local address 127.20.178.66:47331)
0929 03:17:56.952806 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0929 03:17:56.952851 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 03:17:56.953740 (+ 889us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:17:56.953950 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:17:56.953953 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:17:56.954189 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:17:56.954373 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:56.954377 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:56.955003 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:56.955004 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:56.955809 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:56.955811 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:56.956196 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:56.956197 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:56.956291 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:56.956853 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:17:56.956861 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:17:56.958183 (+ 1322us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:17:56.959754 (+ 1571us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:17:56.959761 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:17:56.959763 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:17:56.959976 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:17:56.960230 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:17:56.960231 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:17:56.960231 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:17:56.960359 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:17:56.960699 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:17:56.960706 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:17:56.960839 (+ 133us) client_negotiation.cc:769] Sending connection context
0929 03:17:56.960971 (+ 132us) client_negotiation.cc:240] Negotiation successful
0929 03:17:56.961175 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":82,"threads_started":1}
I20260929 03:17:56.961962 27099 heartbeater.cc:347] Connected to a master server at 127.20.178.126:43487
I20260929 03:17:56.962054 27099 heartbeater.cc:464] Registering TS with master...
I20260929 03:17:56.962239 27099 heartbeater.cc:511] Master 127.20.178.126:43487 requested a full tablet report, sending...
I20260929 03:17:56.962603 26774 ts_manager.cc:194] Registered new tserver with Master: c1317fbe94e6459e8e4e0f0795d5f3a3 (127.20.178.66:38271)
I20260929 03:17:56.962970 26774 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.178.66@KRBTEST.COM'} at 127.20.178.66:47331
WARNING: no policy specified for kudu/127.20.178.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.178.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.178.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.67.keytab.
Entry for principal kudu/127.20.178.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.67.keytab.
I20260929 03:17:56.984946 21193 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuoaHbX/build/release/bin/kudu
/tmp/dist-test-taskuoaHbX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651852476208-21193-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651852476208-21193-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651852476208-21193-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651852476208-21193-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.178.67:0
--local_ip_for_outbound_sockets=127.20.178.67
--webserver_interface=127.20.178.67
--webserver_port=0
--tserver_master_addrs=127.20.178.126:43487
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.67.keytab
--principal=kudu/127.20.178.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.178.84:35359
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651852476208-21193-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651852476208-21193-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651852476208-21193-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:17:57.064718 27104 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:17:57.064916 27104 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:17:57.064939 27104 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:17:57.066007 27104 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:17:57.066039 27104 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:17:57.066064 27104 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.178.67
I20260929 03:17:57.067411 27104 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.178.84:35359
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651852476208-21193-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651852476208-21193-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.67.keytab
--principal=kudu/127.20.178.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.178.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651852476208-21193-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.178.67
--webserver_port=0
--tserver_master_addrs=127.20.178.126:43487
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27104
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.178.67
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651852476208-21193-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:17:57.067584 27104 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:17:57.067759 27104 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:17:57.072585 27109 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:17:57.072613 27112 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:17:57.072661 27110 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:17:57.072875 27104 server_base.cc:1034] running on GCE node
I20260929 03:17:57.073137 27104 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:17:57.073375 27104 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:17:57.074560 27104 hybrid_clock.cc:648] HybridClock initialized: now 1790651877074535 us; error 36 us; skew 500 ppm
Sep 29 03:17:57 dist-test-slave-c9qh krb5kdc[26729](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651877, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:17:57.077144 27104 init.cc:377] Logged in from keytab as kudu/127.20.178.67@KRBTEST.COM (short username kudu)
I20260929 03:17:57.077566 27104 webserver.cc:460] Webserver started at http://127.20.178.67:40267/ using document root <none> and password file <none>
I20260929 03:17:57.077790 27104 fs_manager.cc:359] Metadata directory not provided
I20260929 03:17:57.077858 27104 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:17:57.077988 27104 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:17:57.079031 27104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651852476208-21193-0/minicluster-data/ts-2/data/instance:
uuid: "00a9a02c64d04211af009f915f22842a"
format_stamp: "Formatted at 2026-09-29 03:17:57 on dist-test-slave-c9qh"
server_key: "ec8278fa92c15c0f64f68fe64ed495b3"
server_key_iv: "61b86866dbd4c8dc49f71396ff6bca6b"
server_key_version: "encryptionkey@0"
I20260929 03:17:57.079468 27104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651852476208-21193-0/minicluster-data/ts-2/wal/instance:
uuid: "00a9a02c64d04211af009f915f22842a"
format_stamp: "Formatted at 2026-09-29 03:17:57 on dist-test-slave-c9qh"
server_key: "ec8278fa92c15c0f64f68fe64ed495b3"
server_key_iv: "61b86866dbd4c8dc49f71396ff6bca6b"
server_key_version: "encryptionkey@0"
I20260929 03:17:57.080755 27104 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260929 03:17:57.081473 27119 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:17:57.081629 27104 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 03:17:57.081746 27104 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651852476208-21193-0/minicluster-data/ts-2/data,/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651852476208-21193-0/minicluster-data/ts-2/wal
uuid: "00a9a02c64d04211af009f915f22842a"
format_stamp: "Formatted at 2026-09-29 03:17:57 on dist-test-slave-c9qh"
server_key: "ec8278fa92c15c0f64f68fe64ed495b3"
server_key_iv: "61b86866dbd4c8dc49f71396ff6bca6b"
server_key_version: "encryptionkey@0"
I20260929 03:17:57.081830 27104 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651852476208-21193-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651852476208-21193-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651852476208-21193-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:17:57.092779 27104 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:17:57.094660 27104 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:17:57.094817 27104 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:17:57.095027 27104 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:17:57.095342 27104 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:17:57.095397 27104 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:17:57.095438 27104 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:17:57.095466 27104 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:17:57.101014 27104 rpc_server.cc:307] RPC server started. Bound to: 127.20.178.67:42857
I20260929 03:17:57.101060 27232 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.178.67:42857 every 8 connection(s)
I20260929 03:17:57.101353 27104 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651852476208-21193-0/minicluster-data/ts-2/data/info.pb
Sep 29 03:17:57 dist-test-slave-c9qh krb5kdc[26729](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651877, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.67@KRBTEST.COM for kudu/127.20.178.126@KRBTEST.COM
I20260929 03:17:57.109685 21193 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuoaHbX/build/release/bin/kudu as pid 27104
I20260929 03:17:57.109774 21193 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651852476208-21193-0/minicluster-data/ts-2/wal/instance
I20260929 03:17:57.109979 21193 external_mini_cluster.cc:1428] Setting key c6a852d0b8eb76254edca5cc64febf99
I20260929 03:17:57.110380 26831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:57.102006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.67:40183 (local address 127.20.178.126:43487)
0929 03:17:57.102070 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:57.102072 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:57.102263 (+ 191us) server_negotiation.cc:407] Connection header received
0929 03:17:57.103210 (+ 947us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:57.103211 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:57.103225 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:57.103276 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:17:57.103875 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:57.104374 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:57.105426 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:57.105642 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:57.105731 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:57.107769 (+ 2038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:17:57.107770 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:17:57.107771 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:17:57.107778 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:17:57.109051 (+ 1273us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:57.109399 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:57.109400 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:57.109401 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:57.109449 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:57.109697 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:57.109698 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:57.109699 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:57.109840 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:17:57.109912 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0929 03:17:57.110202 (+ 290us) server_negotiation.cc:299] Negotiation successful
0929 03:17:57.110290 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260929 03:17:57.110587 27235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:57.101967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:43487 (local address 127.20.178.67:40183)
0929 03:17:57.102148 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0929 03:17:57.102183 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0929 03:17:57.103081 (+ 898us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:17:57.103349 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:17:57.103352 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:17:57.103596 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:17:57.103780 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:57.103784 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:57.104471 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:57.104472 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:57.105333 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:57.105335 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:57.105741 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:57.105742 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:57.105863 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:57.106314 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:17:57.106322 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:17:57.107667 (+ 1345us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:17:57.109119 (+ 1452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:17:57.109125 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:17:57.109126 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:17:57.109335 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:17:57.109544 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:17:57.109545 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:17:57.109545 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:17:57.109633 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:17:57.109919 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:17:57.109924 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:17:57.110032 (+ 108us) client_negotiation.cc:769] Sending connection context
0929 03:17:57.110186 (+ 154us) client_negotiation.cc:240] Negotiation successful
0929 03:17:57.110338 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":97,"threads_started":1}
I20260929 03:17:57.111056 27233 heartbeater.cc:347] Connected to a master server at 127.20.178.126:43487
I20260929 03:17:57.111163 27233 heartbeater.cc:464] Registering TS with master...
I20260929 03:17:57.111331 27233 heartbeater.cc:511] Master 127.20.178.126:43487 requested a full tablet report, sending...
I20260929 03:17:57.111699 26773 ts_manager.cc:194] Registered new tserver with Master: 00a9a02c64d04211af009f915f22842a (127.20.178.67:42857)
I20260929 03:17:57.112020 26773 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.178.67@KRBTEST.COM'} at 127.20.178.67:40183
I20260929 03:17:57.121964 21193 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651852476208-21193-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651852476208-21193-0/krb5kdc/test-admin.keytab.
Sep 29 03:17:57 dist-test-slave-c9qh krb5kdc[26729](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651877, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:17:57.138062 21193 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260929 03:17:57.139577 27240 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:17:57.138857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57894 (local address 127.20.178.65:33489)
0929 03:17:57.139010 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:57.139014 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:57.139022 (+ 8us) server_negotiation.cc:407] Connection header received
0929 03:17:57.139045 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:57.139046 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:57.139058 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:57.139141 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:17:57.139476 (+ 335us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57894: BlockingRecv error: recv got EOF from 127.0.0.1:57894 (error 108)
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":61,"threads_started":1}
W20260929 03:17:57.139814 21193 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651852476208-21193-0/krb5kdc/krb5cc' of type FILE
Sep 29 03:17:57 dist-test-slave-c9qh krb5kdc[26729](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651877, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:17:57.140836 21193 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 29 03:17:57 dist-test-slave-c9qh krb5kdc[26729](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651877, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:17:57.141741 21193 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 29 03:17:57 dist-test-slave-c9qh krb5kdc[26729](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651877, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.178.65@KRBTEST.COM
I20260929 03:17:57.149358 27240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:57.142266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57896 (local address 127.20.178.65:33489)
0929 03:17:57.142341 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:57.142343 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:57.142387 (+ 44us) server_negotiation.cc:407] Connection header received
0929 03:17:57.142427 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:57.142427 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:57.142436 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:57.142482 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:17:57.142879 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:57.143588 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:57.144282 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:57.144588 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:57.144748 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:57.146876 (+ 2128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:17:57.146883 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:17:57.146884 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:17:57.146890 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:17:57.148278 (+ 1388us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:57.148581 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:57.148583 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:57.148583 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:57.148694 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:57.148861 (+ 167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:57.148862 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:57.148862 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:57.148983 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:17:57.149076 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0929 03:17:57.149143 (+ 67us) server_negotiation.cc:299] Negotiation successful
0929 03:17:57.149204 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
W20260929 03:17:57.151145 27240 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:17:57.150480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57904 (local address 127.20.178.65:33489)
0929 03:17:57.150542 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:57.150544 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:57.150609 (+ 65us) server_negotiation.cc:407] Connection header received
0929 03:17:57.150735 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:57.150736 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:57.150746 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:57.150801 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:17:57.151092 (+ 291us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57904: BlockingRecv error: recv got EOF from 127.0.0.1:57904 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260929 03:17:57.151218 21193 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651852476208-21193-0/krb5kdc/krb5cc' of type FILE
Sep 29 03:17:57 dist-test-slave-c9qh krb5kdc[26729](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651877, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:17:57.152118 21193 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 29 03:17:57 dist-test-slave-c9qh krb5kdc[26729](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651877, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:17:57.153039 21193 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 29 03:17:57 dist-test-slave-c9qh krb5kdc[26729](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651877, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.178.65@KRBTEST.COM
I20260929 03:17:57.159477 27240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:57.153470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57914 (local address 127.20.178.65:33489)
0929 03:17:57.153525 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:57.153527 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:57.153567 (+ 40us) server_negotiation.cc:407] Connection header received
0929 03:17:57.153636 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:57.153637 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:57.153645 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:57.153721 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:17:57.154065 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:57.154540 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:57.155221 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:57.155423 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:57.155496 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:57.157366 (+ 1870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:17:57.157368 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:17:57.157368 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:17:57.157374 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:17:57.158448 (+ 1074us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:57.158779 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:57.158780 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:57.158780 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:57.158839 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:57.159036 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:57.159037 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:57.159037 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:57.159129 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:17:57.159224 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0929 03:17:57.159327 (+ 103us) server_negotiation.cc:299] Negotiation successful
0929 03:17:57.159393 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
W20260929 03:17:57.160969 27240 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:17:57.160452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57920 (local address 127.20.178.65:33489)
0929 03:17:57.160486 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:57.160489 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:57.160530 (+ 41us) server_negotiation.cc:407] Connection header received
0929 03:17:57.160589 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:57.160590 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:57.160599 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:57.160661 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:17:57.160924 (+ 263us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57920: BlockingRecv error: recv got EOF from 127.0.0.1:57920 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260929 03:17:57.161106 21193 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651852476208-21193-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651852476208-21193-0/krb5kdc/krb5cc)
Sep 29 03:17:57 dist-test-slave-c9qh krb5kdc[26729](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651877, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:17:57.161998 21193 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 29 03:17:57 dist-test-slave-c9qh krb5kdc[26729](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651877, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:17:57.162858 21193 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 29 03:17:57 dist-test-slave-c9qh krb5kdc[26729](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651877, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.178.65@KRBTEST.COM
I20260929 03:17:57.169131 27240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:57.163269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57928 (local address 127.20.178.65:33489)
0929 03:17:57.163318 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:57.163320 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:57.163397 (+ 77us) server_negotiation.cc:407] Connection header received
0929 03:17:57.163431 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:57.163431 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:57.163438 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:57.163480 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:17:57.163804 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:57.164223 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:57.164930 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:57.165097 (+ 167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:57.165146 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:57.166902 (+ 1756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:17:57.166903 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:17:57.166903 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:17:57.166909 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:17:57.168161 (+ 1252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:57.168487 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:57.168488 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:57.168488 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:57.168531 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:57.168736 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:57.168736 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:57.168736 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:57.168815 (+ 79us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:17:57.168876 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0929 03:17:57.168961 (+ 85us) server_negotiation.cc:299] Negotiation successful
0929 03:17:57.169007 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260929 03:17:57.169730 21193 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuoaHbX/build/release/bin/kudu with pid 26836
I20260929 03:17:57.175238 21193 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuoaHbX/build/release/bin/kudu with pid 26970
I20260929 03:17:57.179369 21193 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuoaHbX/build/release/bin/kudu with pid 27104
I20260929 03:17:57.183434 21193 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuoaHbX/build/release/bin/kudu with pid 26742
2026-09-29T03:17:57Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (1866 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:17:57 dist-test-slave-c9qh krb5kdc[27253](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:17:57 dist-test-slave-c9qh krb5kdc[27253](info): set up 2 sockets
Sep 29 03:17:57 dist-test-slave-c9qh krb5kdc[27253](info): commencing operation
krb5kdc: starting...
W20260929 03:17:58.285014 21193 mini_kdc.cc:121] Time spent starting KDC: real 1.082s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/krb5kdc/test-user.keytab.
Sep 29 03:17:58 dist-test-slave-c9qh krb5kdc[27253](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651878, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:17:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:17:58Z Disabled control of system clock
WARNING: no policy specified for oryx/127.20.178.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.20.178.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.20.178.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/krb5kdc/oryx_127.20.178.126.keytab.
Entry for principal oryx/127.20.178.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/krb5kdc/oryx_127.20.178.126.keytab.
I20260929 03:17:58.401119 21193 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuoaHbX/build/release/bin/kudu
/tmp/dist-test-taskuoaHbX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.178.126:35553
--webserver_interface=127.20.178.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/krb5kdc/oryx_127.20.178.126.keytab
--principal=oryx/127.20.178.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.178.84:35629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.178.126:35553
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:17:58.478838 27266 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:17:58.479080 27266 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:17:58.479117 27266 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:17:58.480242 27266 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:17:58.480311 27266 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:17:58.480336 27266 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:17:58.480365 27266 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:17:58.480394 27266 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:17:58.481841 27266 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.178.84:35629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.178.126:35553
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/krb5kdc/oryx_127.20.178.126.keytab
--principal=oryx/127.20.178.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.178.126:35553
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.178.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27266
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:17:58.482100 27266 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:17:58.482324 27266 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:17:58.487010 27271 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:17:58.486999 27274 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:17:58.487104 27266 server_base.cc:1034] running on GCE node
W20260929 03:17:58.487010 27272 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:17:58.487473 27266 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:17:58.487767 27266 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:17:58.488937 27266 hybrid_clock.cc:648] HybridClock initialized: now 1790651878488914 us; error 35 us; skew 500 ppm
Sep 29 03:17:58 dist-test-slave-c9qh krb5kdc[27253](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651878, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.178.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:17:58.491432 27266 init.cc:377] Logged in from keytab as oryx/127.20.178.126@KRBTEST.COM (short username oryx)
I20260929 03:17:58.491881 27266 webserver.cc:460] Webserver started at http://127.20.178.126:36657/ using document root <none> and password file <none>
I20260929 03:17:58.492102 27266 fs_manager.cc:359] Metadata directory not provided
I20260929 03:17:58.492177 27266 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:17:58.492324 27266 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:17:58.493463 27266 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/minicluster-data/master-0/data/instance:
uuid: "d4b1269455e04063b0eb4e2aa46d0684"
format_stamp: "Formatted at 2026-09-29 03:17:58 on dist-test-slave-c9qh"
server_key: "dbc615b79524a48beab8d4ec5d8fa535"
server_key_iv: "7a20f1d82db6e4c1ce57bf87b8b04deb"
server_key_version: "encryptionkey@0"
I20260929 03:17:58.493829 27266 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/minicluster-data/master-0/wal/instance:
uuid: "d4b1269455e04063b0eb4e2aa46d0684"
format_stamp: "Formatted at 2026-09-29 03:17:58 on dist-test-slave-c9qh"
server_key: "dbc615b79524a48beab8d4ec5d8fa535"
server_key_iv: "7a20f1d82db6e4c1ce57bf87b8b04deb"
server_key_version: "encryptionkey@0"
I20260929 03:17:58.495187 27266 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 03:17:58.496079 27281 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:17:58.496256 27266 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 03:17:58.496379 27266 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/minicluster-data/master-0/data,/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/minicluster-data/master-0/wal
uuid: "d4b1269455e04063b0eb4e2aa46d0684"
format_stamp: "Formatted at 2026-09-29 03:17:58 on dist-test-slave-c9qh"
server_key: "dbc615b79524a48beab8d4ec5d8fa535"
server_key_iv: "7a20f1d82db6e4c1ce57bf87b8b04deb"
server_key_version: "encryptionkey@0"
I20260929 03:17:58.496456 27266 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:17:58.509171 27266 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:17:58.511422 27266 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:17:58.511581 27266 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:17:58.515547 27266 rpc_server.cc:307] RPC server started. Bound to: 127.20.178.126:35553
I20260929 03:17:58.515576 27333 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.178.126:35553 every 8 connection(s)
I20260929 03:17:58.515990 27266 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/minicluster-data/master-0/data/info.pb
I20260929 03:17:58.517228 27334 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:17:58.519492 27334 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4b1269455e04063b0eb4e2aa46d0684: Bootstrap starting.
I20260929 03:17:58.520047 27334 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d4b1269455e04063b0eb4e2aa46d0684: Neither blocks nor log segments found. Creating new log.
I20260929 03:17:58.520336 27334 log.cc:824] T 00000000000000000000000000000000 P d4b1269455e04063b0eb4e2aa46d0684: Log is configured to *not* fsync() on all Append() calls
I20260929 03:17:58.521133 27334 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4b1269455e04063b0eb4e2aa46d0684: No bootstrap required, opened a new log
I20260929 03:17:58.522732 27334 raft_consensus.cc:348] T 00000000000000000000000000000000 P d4b1269455e04063b0eb4e2aa46d0684 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4b1269455e04063b0eb4e2aa46d0684" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 35553 } }
I20260929 03:17:58.522866 27334 raft_consensus.cc:374] T 00000000000000000000000000000000 P d4b1269455e04063b0eb4e2aa46d0684 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:17:58.522886 27334 raft_consensus.cc:729] T 00000000000000000000000000000000 P d4b1269455e04063b0eb4e2aa46d0684 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4b1269455e04063b0eb4e2aa46d0684, State: Initialized, Role: FOLLOWER
I20260929 03:17:58.523067 27334 consensus_queue.cc:260] T 00000000000000000000000000000000 P d4b1269455e04063b0eb4e2aa46d0684 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4b1269455e04063b0eb4e2aa46d0684" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 35553 } }
I20260929 03:17:58.523130 27334 raft_consensus.cc:388] T 00000000000000000000000000000000 P d4b1269455e04063b0eb4e2aa46d0684 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:17:58.523152 27334 raft_consensus.cc:482] T 00000000000000000000000000000000 P d4b1269455e04063b0eb4e2aa46d0684 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:17:58.523223 27334 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d4b1269455e04063b0eb4e2aa46d0684 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:17:58.524029 27334 raft_consensus.cc:504] T 00000000000000000000000000000000 P d4b1269455e04063b0eb4e2aa46d0684 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4b1269455e04063b0eb4e2aa46d0684" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 35553 } }
I20260929 03:17:58.524114 27334 leader_election.cc:302] T 00000000000000000000000000000000 P d4b1269455e04063b0eb4e2aa46d0684 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d4b1269455e04063b0eb4e2aa46d0684; no voters:
I20260929 03:17:58.524296 27334 leader_election.cc:288] T 00000000000000000000000000000000 P d4b1269455e04063b0eb4e2aa46d0684 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:17:58.524334 27337 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d4b1269455e04063b0eb4e2aa46d0684 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:17:58.524468 27337 raft_consensus.cc:686] T 00000000000000000000000000000000 P d4b1269455e04063b0eb4e2aa46d0684 [term 1 LEADER]: Becoming Leader. State: Replica: d4b1269455e04063b0eb4e2aa46d0684, State: Running, Role: LEADER
I20260929 03:17:58.524550 27334 sys_catalog.cc:564] T 00000000000000000000000000000000 P d4b1269455e04063b0eb4e2aa46d0684 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:17:58.524603 27337 consensus_queue.cc:237] T 00000000000000000000000000000000 P d4b1269455e04063b0eb4e2aa46d0684 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4b1269455e04063b0eb4e2aa46d0684" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 35553 } }
I20260929 03:17:58.525002 27339 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4b1269455e04063b0eb4e2aa46d0684 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d4b1269455e04063b0eb4e2aa46d0684. Latest consensus state: current_term: 1 leader_uuid: "d4b1269455e04063b0eb4e2aa46d0684" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4b1269455e04063b0eb4e2aa46d0684" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 35553 } } }
I20260929 03:17:58.525010 27338 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4b1269455e04063b0eb4e2aa46d0684 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d4b1269455e04063b0eb4e2aa46d0684" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4b1269455e04063b0eb4e2aa46d0684" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 35553 } } }
I20260929 03:17:58.525106 27339 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4b1269455e04063b0eb4e2aa46d0684 [sys.catalog]: This master's current role is: LEADER
I20260929 03:17:58.525112 27338 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4b1269455e04063b0eb4e2aa46d0684 [sys.catalog]: This master's current role is: LEADER
I20260929 03:17:58.525344 27346 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:17:58.525355 21193 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuoaHbX/build/release/bin/kudu as pid 27266
I20260929 03:17:58.525447 21193 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/minicluster-data/master-0/wal/instance
I20260929 03:17:58.525645 21193 external_mini_cluster.cc:1428] Setting key f1ec3f9dbf0e8ea1c092fec677a58f1f
I20260929 03:17:58.526081 27346 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:17:58.527760 27346 catalog_manager.cc:1347] Generated new cluster ID: d4b12290c9754a30a1ed0ccfd1dc774e
I20260929 03:17:58.527809 27346 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 29 03:17:58 dist-test-slave-c9qh krb5kdc[27253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651878, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.20.178.126@KRBTEST.COM
I20260929 03:17:58.535656 27356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:58.526298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42420 (local address 127.20.178.126:35553)
0929 03:17:58.526462 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:58.526473 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:58.526628 (+ 155us) server_negotiation.cc:407] Connection header received
0929 03:17:58.526791 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:58.526799 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:58.526856 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:58.527006 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:17:58.527783 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:58.528896 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:58.529645 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:58.530064 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:58.530195 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:58.532590 (+ 2395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:17:58.532593 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:17:58.532594 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:17:58.532605 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:17:58.534135 (+ 1530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:58.534511 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:58.534513 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:58.534514 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:58.534611 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:58.534828 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:58.534829 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:58.534829 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:58.535052 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:17:58.535207 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0929 03:17:58.535317 (+ 110us) server_negotiation.cc:299] Negotiation successful
0929 03:17:58.535454 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":75,"threads_started":1}
I20260929 03:17:58.536070 27346 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:17:58.536672 27346 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:17:58.542030 27346 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d4b1269455e04063b0eb4e2aa46d0684: Generated new TSK 0
I20260929 03:17:58.542201 27346 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.20.178.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.20.178.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.20.178.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/krb5kdc/oryx_127.20.178.65.keytab.
Entry for principal oryx/127.20.178.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/krb5kdc/oryx_127.20.178.65.keytab.
I20260929 03:17:58.573378 21193 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuoaHbX/build/release/bin/kudu
/tmp/dist-test-taskuoaHbX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.178.65:0
--local_ip_for_outbound_sockets=127.20.178.65
--webserver_interface=127.20.178.65
--webserver_port=0
--tserver_master_addrs=127.20.178.126:35553
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/krb5kdc/oryx_127.20.178.65.keytab
--principal=oryx/127.20.178.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.178.84:35629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:17:58.676290 27360 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:17:58.676514 27360 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:17:58.676550 27360 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:17:58.677721 27360 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:17:58.677790 27360 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:17:58.677832 27360 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.178.65
I20260929 03:17:58.679291 27360 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.178.84:35629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/krb5kdc/oryx_127.20.178.65.keytab
--principal=oryx/127.20.178.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.178.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.178.65
--webserver_port=0
--tserver_master_addrs=127.20.178.126:35553
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27360
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.178.65
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:17:58.679550 27360 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:17:58.679776 27360 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:17:58.684793 27366 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:17:58.684825 27368 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:17:58.684863 27365 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:17:58.685014 27360 server_base.cc:1034] running on GCE node
I20260929 03:17:58.685281 27360 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:17:58.685513 27360 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:17:58.686651 27360 hybrid_clock.cc:648] HybridClock initialized: now 1790651878686628 us; error 40 us; skew 500 ppm
Sep 29 03:17:58 dist-test-slave-c9qh krb5kdc[27253](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651878, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.178.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:17:58.689360 27360 init.cc:377] Logged in from keytab as oryx/127.20.178.65@KRBTEST.COM (short username oryx)
I20260929 03:17:58.689774 27360 webserver.cc:460] Webserver started at http://127.20.178.65:36929/ using document root <none> and password file <none>
I20260929 03:17:58.689954 27360 fs_manager.cc:359] Metadata directory not provided
I20260929 03:17:58.690032 27360 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:17:58.690158 27360 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:17:58.691198 27360 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/minicluster-data/ts-0/data/instance:
uuid: "e76cae1a85be4646b41cc2ed226fe5c7"
format_stamp: "Formatted at 2026-09-29 03:17:58 on dist-test-slave-c9qh"
server_key: "15d4b51628ea93bffd5ebff7517c9500"
server_key_iv: "3c2e60dfdc4fc18b3813c00ab4e59740"
server_key_version: "encryptionkey@0"
I20260929 03:17:58.691547 27360 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/minicluster-data/ts-0/wal/instance:
uuid: "e76cae1a85be4646b41cc2ed226fe5c7"
format_stamp: "Formatted at 2026-09-29 03:17:58 on dist-test-slave-c9qh"
server_key: "15d4b51628ea93bffd5ebff7517c9500"
server_key_iv: "3c2e60dfdc4fc18b3813c00ab4e59740"
server_key_version: "encryptionkey@0"
I20260929 03:17:58.692970 27360 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 03:17:58.693763 27375 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:17:58.693936 27360 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260929 03:17:58.694034 27360 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/minicluster-data/ts-0/data,/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/minicluster-data/ts-0/wal
uuid: "e76cae1a85be4646b41cc2ed226fe5c7"
format_stamp: "Formatted at 2026-09-29 03:17:58 on dist-test-slave-c9qh"
server_key: "15d4b51628ea93bffd5ebff7517c9500"
server_key_iv: "3c2e60dfdc4fc18b3813c00ab4e59740"
server_key_version: "encryptionkey@0"
I20260929 03:17:58.694113 27360 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:17:58.715204 27360 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:17:58.716984 27360 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:17:58.717145 27360 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:17:58.717365 27360 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:17:58.717672 27360 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:17:58.717728 27360 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:17:58.717760 27360 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:17:58.717788 27360 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:17:58.723400 27360 rpc_server.cc:307] RPC server started. Bound to: 127.20.178.65:45139
I20260929 03:17:58.723471 27488 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.178.65:45139 every 8 connection(s)
I20260929 03:17:58.723781 27360 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/minicluster-data/ts-0/data/info.pb
I20260929 03:17:58.728549 21193 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuoaHbX/build/release/bin/kudu as pid 27360
I20260929 03:17:58.728672 21193 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/minicluster-data/ts-0/wal/instance
I20260929 03:17:58.728871 21193 external_mini_cluster.cc:1428] Setting key 3ffe9f3c02c0b995d77495dd7b56bf2a
Sep 29 03:17:58 dist-test-slave-c9qh krb5kdc[27253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651878, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.178.65@KRBTEST.COM for oryx/127.20.178.126@KRBTEST.COM
I20260929 03:17:58.733147 27356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:58.724531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.65:38129 (local address 127.20.178.126:35553)
0929 03:17:58.724590 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:58.724592 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:58.724827 (+ 235us) server_negotiation.cc:407] Connection header received
0929 03:17:58.725376 (+ 549us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:58.725377 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:58.725389 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:58.725448 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:17:58.726375 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:58.726874 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:58.727804 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:58.727993 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:58.728044 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:58.730233 (+ 2189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:17:58.730235 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:17:58.730235 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:17:58.730242 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:17:58.731702 (+ 1460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:58.732090 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:58.732092 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:58.732092 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:58.732157 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:58.732412 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:58.732412 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:58.732413 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:58.732543 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:17:58.732650 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0929 03:17:58.732952 (+ 302us) server_negotiation.cc:299] Negotiation successful
0929 03:17:58.733047 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260929 03:17:58.733283 27491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:58.724490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:35553 (local address 127.20.178.65:38129)
0929 03:17:58.724679 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0929 03:17:58.724722 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 03:17:58.725221 (+ 499us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:17:58.725506 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:17:58.725509 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:17:58.726114 (+ 605us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:17:58.726298 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:58.726303 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:58.726952 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:58.726953 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:58.727730 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:58.727732 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:58.728065 (+ 333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:58.728065 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:58.728165 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:58.728766 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:17:58.728774 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:17:58.730139 (+ 1365us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:17:58.731772 (+ 1633us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:17:58.731777 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:17:58.731779 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:17:58.731982 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:17:58.732236 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:17:58.732237 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:17:58.732237 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:17:58.732338 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:17:58.732676 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:17:58.732683 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:17:58.732809 (+ 126us) client_negotiation.cc:769] Sending connection context
0929 03:17:58.732917 (+ 108us) client_negotiation.cc:240] Negotiation successful
0929 03:17:58.733066 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":98,"threads_started":1}
I20260929 03:17:58.733776 27489 heartbeater.cc:347] Connected to a master server at 127.20.178.126:35553
I20260929 03:17:58.733880 27489 heartbeater.cc:464] Registering TS with master...
I20260929 03:17:58.734079 27489 heartbeater.cc:511] Master 127.20.178.126:35553 requested a full tablet report, sending...
I20260929 03:17:58.734544 27298 ts_manager.cc:194] Registered new tserver with Master: e76cae1a85be4646b41cc2ed226fe5c7 (127.20.178.65:45139)
I20260929 03:17:58.735174 27298 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.20.178.65@KRBTEST.COM'} at 127.20.178.65:38129
WARNING: no policy specified for oryx/127.20.178.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.20.178.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.20.178.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/krb5kdc/oryx_127.20.178.66.keytab.
Entry for principal oryx/127.20.178.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/krb5kdc/oryx_127.20.178.66.keytab.
I20260929 03:17:58.753599 21193 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuoaHbX/build/release/bin/kudu
/tmp/dist-test-taskuoaHbX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.178.66:0
--local_ip_for_outbound_sockets=127.20.178.66
--webserver_interface=127.20.178.66
--webserver_port=0
--tserver_master_addrs=127.20.178.126:35553
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/krb5kdc/oryx_127.20.178.66.keytab
--principal=oryx/127.20.178.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.178.84:35629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:17:58.832831 27494 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:17:58.833019 27494 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:17:58.833041 27494 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:17:58.834188 27494 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:17:58.834232 27494 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:17:58.834259 27494 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.178.66
I20260929 03:17:58.835668 27494 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.178.84:35629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/krb5kdc/oryx_127.20.178.66.keytab
--principal=oryx/127.20.178.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.178.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.178.66
--webserver_port=0
--tserver_master_addrs=127.20.178.126:35553
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27494
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.178.66
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:17:58.835908 27494 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:17:58.836145 27494 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:17:58.841398 27499 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:17:58.841403 27500 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:17:58.841634 27494 server_base.cc:1034] running on GCE node
W20260929 03:17:58.841410 27502 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:17:58.841926 27494 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:17:58.842186 27494 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:17:58.843338 27494 hybrid_clock.cc:648] HybridClock initialized: now 1790651878843328 us; error 32 us; skew 500 ppm
Sep 29 03:17:58 dist-test-slave-c9qh krb5kdc[27253](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651878, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.178.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:17:58.845994 27494 init.cc:377] Logged in from keytab as oryx/127.20.178.66@KRBTEST.COM (short username oryx)
I20260929 03:17:58.846472 27494 webserver.cc:460] Webserver started at http://127.20.178.66:41939/ using document root <none> and password file <none>
I20260929 03:17:58.846763 27494 fs_manager.cc:359] Metadata directory not provided
I20260929 03:17:58.846835 27494 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:17:58.846994 27494 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:17:58.848120 27494 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/minicluster-data/ts-1/data/instance:
uuid: "5064f9a44a034c0dadb4054ac8824780"
format_stamp: "Formatted at 2026-09-29 03:17:58 on dist-test-slave-c9qh"
server_key: "eed6d3c0ad3474a10b2e7d435b78af62"
server_key_iv: "f24a1a875cb2a1c09df321bdff4c6df8"
server_key_version: "encryptionkey@0"
I20260929 03:17:58.848500 27494 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/minicluster-data/ts-1/wal/instance:
uuid: "5064f9a44a034c0dadb4054ac8824780"
format_stamp: "Formatted at 2026-09-29 03:17:58 on dist-test-slave-c9qh"
server_key: "eed6d3c0ad3474a10b2e7d435b78af62"
server_key_iv: "f24a1a875cb2a1c09df321bdff4c6df8"
server_key_version: "encryptionkey@0"
I20260929 03:17:58.849880 27494 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 03:17:58.850618 27509 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:17:58.850798 27494 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 03:17:58.850896 27494 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/minicluster-data/ts-1/data,/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/minicluster-data/ts-1/wal
uuid: "5064f9a44a034c0dadb4054ac8824780"
format_stamp: "Formatted at 2026-09-29 03:17:58 on dist-test-slave-c9qh"
server_key: "eed6d3c0ad3474a10b2e7d435b78af62"
server_key_iv: "f24a1a875cb2a1c09df321bdff4c6df8"
server_key_version: "encryptionkey@0"
I20260929 03:17:58.850980 27494 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:17:58.870129 27494 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:17:58.872080 27494 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:17:58.872246 27494 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:17:58.872470 27494 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:17:58.872871 27494 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:17:58.872929 27494 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:17:58.872969 27494 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:17:58.872997 27494 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:17:58.878952 27494 rpc_server.cc:307] RPC server started. Bound to: 127.20.178.66:42401
I20260929 03:17:58.878995 27622 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.178.66:42401 every 8 connection(s)
I20260929 03:17:58.879307 27494 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/minicluster-data/ts-1/data/info.pb
Sep 29 03:17:58 dist-test-slave-c9qh krb5kdc[27253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651878, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.178.66@KRBTEST.COM for oryx/127.20.178.126@KRBTEST.COM
I20260929 03:17:58.888167 21193 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuoaHbX/build/release/bin/kudu as pid 27494
I20260929 03:17:58.888286 21193 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/minicluster-data/ts-1/wal/instance
I20260929 03:17:58.888424 27356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:58.880034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.66:56315 (local address 127.20.178.126:35553)
0929 03:17:58.880099 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:58.880101 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:58.880308 (+ 207us) server_negotiation.cc:407] Connection header received
0929 03:17:58.880871 (+ 563us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:58.880873 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:58.880883 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:58.880969 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:17:58.881855 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:58.882356 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:58.883346 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:58.883538 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:58.883680 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:58.885731 (+ 2051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:17:58.885732 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:17:58.885732 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:17:58.885739 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:17:58.886967 (+ 1228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:58.887332 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:58.887333 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:58.887333 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:58.887381 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:58.887665 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:58.887666 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:58.887667 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:58.887806 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:17:58.887871 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0929 03:17:58.888254 (+ 383us) server_negotiation.cc:299] Negotiation successful
0929 03:17:58.888351 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260929 03:17:58.888489 21193 external_mini_cluster.cc:1428] Setting key c4fcf9ea871e5e8b2104576971528548
I20260929 03:17:58.888626 27625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:58.880019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:35553 (local address 127.20.178.66:56315)
0929 03:17:58.880171 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0929 03:17:58.880219 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0929 03:17:58.880760 (+ 541us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:17:58.881025 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:17:58.881028 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:17:58.881586 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:17:58.881779 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:58.881783 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:58.882431 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:58.882431 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:58.883249 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:58.883251 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:58.883672 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:58.883672 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:58.883769 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:58.884236 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:17:58.884244 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:17:58.885630 (+ 1386us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:17:58.887036 (+ 1406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:17:58.887042 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:17:58.887043 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:17:58.887267 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:17:58.887457 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:17:58.887458 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:17:58.887458 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:17:58.887575 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:17:58.887872 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:17:58.887876 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:17:58.888072 (+ 196us) client_negotiation.cc:769] Sending connection context
0929 03:17:58.888202 (+ 130us) client_negotiation.cc:240] Negotiation successful
0929 03:17:58.888380 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":83,"threads_started":1}
I20260929 03:17:58.889277 27623 heartbeater.cc:347] Connected to a master server at 127.20.178.126:35553
I20260929 03:17:58.889389 27623 heartbeater.cc:464] Registering TS with master...
I20260929 03:17:58.889569 27623 heartbeater.cc:511] Master 127.20.178.126:35553 requested a full tablet report, sending...
I20260929 03:17:58.889961 27298 ts_manager.cc:194] Registered new tserver with Master: 5064f9a44a034c0dadb4054ac8824780 (127.20.178.66:42401)
I20260929 03:17:58.890439 27298 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.20.178.66@KRBTEST.COM'} at 127.20.178.66:56315
WARNING: no policy specified for oryx/127.20.178.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.20.178.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.20.178.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/krb5kdc/oryx_127.20.178.67.keytab.
Entry for principal oryx/127.20.178.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/krb5kdc/oryx_127.20.178.67.keytab.
I20260929 03:17:58.914871 21193 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuoaHbX/build/release/bin/kudu
/tmp/dist-test-taskuoaHbX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.178.67:0
--local_ip_for_outbound_sockets=127.20.178.67
--webserver_interface=127.20.178.67
--webserver_port=0
--tserver_master_addrs=127.20.178.126:35553
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/krb5kdc/oryx_127.20.178.67.keytab
--principal=oryx/127.20.178.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.178.84:35629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:17:58.993800 27628 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:17:58.993985 27628 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:17:58.994007 27628 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:17:58.995131 27628 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:17:58.995170 27628 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:17:58.995196 27628 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.178.67
I20260929 03:17:58.996620 27628 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.178.84:35629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/krb5kdc/oryx_127.20.178.67.keytab
--principal=oryx/127.20.178.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.178.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.178.67
--webserver_port=0
--tserver_master_addrs=127.20.178.126:35553
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27628
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.178.67
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:17:58.996878 27628 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:17:58.997128 27628 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:17:59.001878 27634 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:17:59.001915 27633 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:17:59.001916 27636 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:17:59.002081 27628 server_base.cc:1034] running on GCE node
I20260929 03:17:59.002367 27628 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:17:59.002619 27628 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:17:59.003767 27628 hybrid_clock.cc:648] HybridClock initialized: now 1790651879003761 us; error 29 us; skew 500 ppm
Sep 29 03:17:59 dist-test-slave-c9qh krb5kdc[27253](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651879, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.178.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:17:59.006178 27628 init.cc:377] Logged in from keytab as oryx/127.20.178.67@KRBTEST.COM (short username oryx)
I20260929 03:17:59.006660 27628 webserver.cc:460] Webserver started at http://127.20.178.67:40205/ using document root <none> and password file <none>
I20260929 03:17:59.006891 27628 fs_manager.cc:359] Metadata directory not provided
I20260929 03:17:59.006956 27628 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:17:59.007105 27628 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:17:59.008122 27628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/minicluster-data/ts-2/data/instance:
uuid: "b2e7ad7ddf214b3596c571f5cd4d3272"
format_stamp: "Formatted at 2026-09-29 03:17:59 on dist-test-slave-c9qh"
server_key: "a1b2520717078d0eb5d17adc8311d1a5"
server_key_iv: "4bab638e4583baef20b315406a7b38cf"
server_key_version: "encryptionkey@0"
I20260929 03:17:59.008482 27628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/minicluster-data/ts-2/wal/instance:
uuid: "b2e7ad7ddf214b3596c571f5cd4d3272"
format_stamp: "Formatted at 2026-09-29 03:17:59 on dist-test-slave-c9qh"
server_key: "a1b2520717078d0eb5d17adc8311d1a5"
server_key_iv: "4bab638e4583baef20b315406a7b38cf"
server_key_version: "encryptionkey@0"
I20260929 03:17:59.009809 27628 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260929 03:17:59.010533 27643 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:17:59.010771 27628 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260929 03:17:59.010886 27628 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/minicluster-data/ts-2/data,/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/minicluster-data/ts-2/wal
uuid: "b2e7ad7ddf214b3596c571f5cd4d3272"
format_stamp: "Formatted at 2026-09-29 03:17:59 on dist-test-slave-c9qh"
server_key: "a1b2520717078d0eb5d17adc8311d1a5"
server_key_iv: "4bab638e4583baef20b315406a7b38cf"
server_key_version: "encryptionkey@0"
I20260929 03:17:59.010972 27628 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:17:59.020164 27628 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:17:59.022082 27628 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:17:59.022235 27628 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:17:59.022436 27628 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:17:59.022732 27628 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:17:59.022796 27628 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:17:59.022842 27628 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:17:59.022864 27628 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:17:59.028946 27628 rpc_server.cc:307] RPC server started. Bound to: 127.20.178.67:32833
I20260929 03:17:59.029003 27756 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.178.67:32833 every 8 connection(s)
I20260929 03:17:59.029294 27628 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/minicluster-data/ts-2/data/info.pb
Sep 29 03:17:59 dist-test-slave-c9qh krb5kdc[27253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651879, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.178.67@KRBTEST.COM for oryx/127.20.178.126@KRBTEST.COM
I20260929 03:17:59.038359 27356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:59.029933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.67:41771 (local address 127.20.178.126:35553)
0929 03:17:59.030010 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:59.030048 (+ 38us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:59.030237 (+ 189us) server_negotiation.cc:407] Connection header received
0929 03:17:59.030755 (+ 518us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:59.030756 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:59.030770 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:59.030823 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:17:59.031782 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:59.032284 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:59.033360 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:59.033573 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:59.033661 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:59.035774 (+ 2113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:17:59.035776 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:17:59.035776 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:17:59.035784 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:17:59.037071 (+ 1287us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:59.037528 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:59.037529 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:59.037529 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:59.037594 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:59.037845 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:59.037846 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:59.037846 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:59.037947 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:17:59.038001 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0929 03:17:59.038239 (+ 238us) server_negotiation.cc:299] Negotiation successful
0929 03:17:59.038298 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 03:17:59.038537 27759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:59.029924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:35553 (local address 127.20.178.67:41771)
0929 03:17:59.030114 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0929 03:17:59.030153 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 03:17:59.030650 (+ 497us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:17:59.030899 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:17:59.030902 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:17:59.031446 (+ 544us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:17:59.031676 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:59.031681 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:59.032376 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:59.032378 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:59.033254 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:59.033256 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:59.033669 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:59.033670 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:59.033795 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:59.034302 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:17:59.034310 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:17:59.035674 (+ 1364us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:17:59.037210 (+ 1536us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:17:59.037217 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:17:59.037219 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:17:59.037435 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:17:59.037672 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:17:59.037673 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:17:59.037673 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:17:59.037776 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:17:59.038008 (+ 232us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:17:59.038011 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:17:59.038128 (+ 117us) client_negotiation.cc:769] Sending connection context
0929 03:17:59.038208 (+ 80us) client_negotiation.cc:240] Negotiation successful
0929 03:17:59.038331 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":77,"threads_started":1}
I20260929 03:17:59.039093 27757 heartbeater.cc:347] Connected to a master server at 127.20.178.126:35553
I20260929 03:17:59.039197 27757 heartbeater.cc:464] Registering TS with master...
I20260929 03:17:59.039203 21193 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuoaHbX/build/release/bin/kudu as pid 27628
I20260929 03:17:59.039276 21193 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/minicluster-data/ts-2/wal/instance
I20260929 03:17:59.039387 27757 heartbeater.cc:511] Master 127.20.178.126:35553 requested a full tablet report, sending...
I20260929 03:17:59.039454 21193 external_mini_cluster.cc:1428] Setting key 8b98782d3d2da7249ffb50f6a93bfb8f
I20260929 03:17:59.039772 27298 ts_manager.cc:194] Registered new tserver with Master: b2e7ad7ddf214b3596c571f5cd4d3272 (127.20.178.67:32833)
I20260929 03:17:59.040238 27298 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.20.178.67@KRBTEST.COM'} at 127.20.178.67:41771
I20260929 03:17:59.040479 21193 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.20.178.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.20.178.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.20.178.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/krb5kdc/oryx_127.20.178.125.keytab.
Entry for principal oryx/127.20.178.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/krb5kdc/oryx_127.20.178.125.keytab.
I20260929 03:17:59.063414 21193 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuoaHbX/build/release/bin/kudu
/tmp/dist-test-taskuoaHbX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.178.125:35183
--webserver_interface=127.20.178.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/krb5kdc/oryx_127.20.178.125.keytab
--principal=oryx/127.20.178.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.178.84:35629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.178.126:35553,127.20.178.125:35183
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:17:59.141813 27762 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:17:59.142030 27762 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:17:59.142067 27762 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:17:59.143193 27762 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:17:59.143257 27762 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:17:59.143285 27762 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:17:59.143316 27762 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:17:59.143338 27762 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:17:59.144822 27762 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.178.84:35629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.20.178.126:35553,127.20.178.125:35183
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/krb5kdc/oryx_127.20.178.125.keytab
--principal=oryx/127.20.178.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.178.125:35183
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.20.178.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27762
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
W20260929 03:17:59.144951 27762 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.20.178.126:35553,127.20.178.125:35183'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260929 03:17:59.145098 27762 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:17:59.145308 27762 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:17:59.150043 27768 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:17:59.150070 27767 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:17:59.150039 27770 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:17:59.150445 27762 server_base.cc:1034] running on GCE node
I20260929 03:17:59.150713 27762 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:17:59.150976 27762 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:17:59.152117 27762 hybrid_clock.cc:648] HybridClock initialized: now 1790651879152098 us; error 36 us; skew 500 ppm
Sep 29 03:17:59 dist-test-slave-c9qh krb5kdc[27253](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651879, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.178.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:17:59.154497 27762 init.cc:377] Logged in from keytab as oryx/127.20.178.125@KRBTEST.COM (short username oryx)
I20260929 03:17:59.155030 27762 webserver.cc:460] Webserver started at http://127.20.178.125:41883/ using document root <none> and password file <none>
I20260929 03:17:59.155257 27762 fs_manager.cc:359] Metadata directory not provided
I20260929 03:17:59.155321 27762 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:17:59.155476 27762 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:17:59.156451 27762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/minicluster-data/master-1/data/instance:
uuid: "82e1ecef80f449029732e3ade3c4a092"
format_stamp: "Formatted at 2026-09-29 03:17:59 on dist-test-slave-c9qh"
server_key: "612b8ffb623b4a6b7a15f71234bdde3d"
server_key_iv: "444ce762e4607baaa494fcc4968d2523"
server_key_version: "encryptionkey@0"
I20260929 03:17:59.156823 27762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/minicluster-data/master-1/wal/instance:
uuid: "82e1ecef80f449029732e3ade3c4a092"
format_stamp: "Formatted at 2026-09-29 03:17:59 on dist-test-slave-c9qh"
server_key: "612b8ffb623b4a6b7a15f71234bdde3d"
server_key_iv: "444ce762e4607baaa494fcc4968d2523"
server_key_version: "encryptionkey@0"
I20260929 03:17:59.158114 27762 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260929 03:17:59.158813 27777 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:17:59.158989 27762 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 03:17:59.159081 27762 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/minicluster-data/master-1/data,/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/minicluster-data/master-1/wal
uuid: "82e1ecef80f449029732e3ade3c4a092"
format_stamp: "Formatted at 2026-09-29 03:17:59 on dist-test-slave-c9qh"
server_key: "612b8ffb623b4a6b7a15f71234bdde3d"
server_key_iv: "444ce762e4607baaa494fcc4968d2523"
server_key_version: "encryptionkey@0"
I20260929 03:17:59.159164 27762 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:17:59.184593 27762 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:17:59.186388 27762 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:17:59.186546 27762 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:17:59.190353 27762 rpc_server.cc:307] RPC server started. Bound to: 127.20.178.125:35183
I20260929 03:17:59.190382 27829 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.178.125:35183 every 8 connection(s)
I20260929 03:17:59.190711 27762 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/minicluster-data/master-1/data/info.pb
I20260929 03:17:59.191721 27830 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:17:59.193218 27830 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.178.126" port: 35553 } has no permanent_uuid. Determining permanent_uuid...
I20260929 03:17:59.197742 21193 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuoaHbX/build/release/bin/kudu as pid 27762
Sep 29 03:17:59 dist-test-slave-c9qh krb5kdc[27253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651879, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.178.125@KRBTEST.COM for oryx/127.20.178.126@KRBTEST.COM
I20260929 03:17:59.202239 27356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:59.193966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42432 (local address 127.20.178.126:35553)
0929 03:17:59.194034 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:59.194036 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:59.194199 (+ 163us) server_negotiation.cc:407] Connection header received
0929 03:17:59.194612 (+ 413us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:59.194613 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:59.194622 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:59.194689 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:17:59.195233 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:59.195742 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:59.196844 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:59.197072 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:59.197127 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:59.199468 (+ 2341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:17:59.199470 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:17:59.199470 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:17:59.199476 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:17:59.200892 (+ 1416us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:59.201278 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:59.201279 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:59.201280 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:59.201336 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:59.201583 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:59.201584 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:59.201584 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:59.201730 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:17:59.201807 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0929 03:17:59.202066 (+ 259us) server_negotiation.cc:299] Negotiation successful
0929 03:17:59.202141 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260929 03:17:59.202407 27832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:59.193928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:35553 (local address 127.0.0.1:42432)
0929 03:17:59.194076 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0929 03:17:59.194114 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 03:17:59.194535 (+ 421us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:17:59.194741 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:17:59.194745 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:17:59.194969 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:17:59.195151 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:59.195155 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:59.195818 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:59.195818 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:59.196719 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:59.196722 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:59.197141 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:59.197142 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:59.197253 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:59.197775 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:17:59.197783 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:17:59.199386 (+ 1603us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:17:59.200977 (+ 1591us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:17:59.200983 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:17:59.200985 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:17:59.201203 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:17:59.201404 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:17:59.201405 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:17:59.201405 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:17:59.201514 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:17:59.201808 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:17:59.201814 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:17:59.201943 (+ 129us) client_negotiation.cc:769] Sending connection context
0929 03:17:59.202045 (+ 102us) client_negotiation.cc:240] Negotiation successful
0929 03:17:59.202187 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":79,"threads_started":1}
I20260929 03:17:59.203073 27830 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.178.125" port: 35183 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 03:17:59 dist-test-slave-c9qh krb5kdc[27253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651878, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.20.178.125@KRBTEST.COM
I20260929 03:17:59.206794 27834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:59.198264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54130 (local address 127.20.178.125:35183)
0929 03:17:59.198457 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:59.198461 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:59.198481 (+ 20us) server_negotiation.cc:407] Connection header received
0929 03:17:59.198556 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:59.198558 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:59.198578 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:59.198667 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:17:59.199345 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:59.200377 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:59.201160 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:59.201475 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:59.201557 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:59.204202 (+ 2645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:17:59.204210 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:17:59.204210 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:17:59.204216 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:17:59.205626 (+ 1410us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:59.205974 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:59.205978 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:59.205978 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:59.206037 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:59.206248 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:59.206249 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:59.206249 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:59.206380 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:17:59.206533 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0929 03:17:59.206615 (+ 82us) server_negotiation.cc:299] Negotiation successful
0929 03:17:59.206688 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":96,"threads_started":1}
Sep 29 03:17:59 dist-test-slave-c9qh krb5kdc[27253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651879, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.178.125@KRBTEST.COM for oryx/127.20.178.125@KRBTEST.COM
I20260929 03:17:59.209915 27832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:59.203299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.125:35183 (local address 127.0.0.1:54134)
0929 03:17:59.203346 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0929 03:17:59.203356 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 03:17:59.203394 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:17:59.203621 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:17:59.203622 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:17:59.203732 (+ 110us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:17:59.203776 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:59.203798 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:59.204678 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:59.204679 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:59.205304 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:59.205306 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:59.205693 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:59.205694 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:59.205757 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:59.206113 (+ 356us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:17:59.206117 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:17:59.207320 (+ 1203us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:17:59.209206 (+ 1886us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:17:59.209207 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:17:59.209207 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:17:59.209357 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:17:59.209519 (+ 162us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:17:59.209520 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:17:59.209520 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:17:59.209565 (+ 45us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:17:59.209722 (+ 157us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:17:59.209722 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:17:59.209750 (+ 28us) client_negotiation.cc:769] Sending connection context
0929 03:17:59.209795 (+ 45us) client_negotiation.cc:240] Negotiation successful
0929 03:17:59.209859 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260929 03:17:59.210022 27835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:59.203349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54134 (local address 127.20.178.125:35183)
0929 03:17:59.203498 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:59.203500 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:59.203509 (+ 9us) server_negotiation.cc:407] Connection header received
0929 03:17:59.203528 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:59.203529 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:59.203540 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:59.203619 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:17:59.203875 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:59.204351 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:59.205402 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:59.205592 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:59.205676 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:59.207394 (+ 1718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:17:59.207396 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:17:59.207396 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:17:59.207403 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:17:59.209133 (+ 1730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:59.209418 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:59.209418 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:59.209418 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:59.209465 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:59.209602 (+ 137us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:59.209602 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:59.209602 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:59.209680 (+ 78us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:17:59.209723 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0929 03:17:59.209808 (+ 85us) server_negotiation.cc:299] Negotiation successful
0929 03:17:59.209902 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":40,"server-negotiator.queue_time_us":128,"thread_start_us":86,"threads_started":1}
I20260929 03:17:59.211149 27830 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82e1ecef80f449029732e3ade3c4a092: Bootstrap starting.
I20260929 03:17:59.211743 27830 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 82e1ecef80f449029732e3ade3c4a092: Neither blocks nor log segments found. Creating new log.
I20260929 03:17:59.212007 27830 log.cc:824] T 00000000000000000000000000000000 P 82e1ecef80f449029732e3ade3c4a092: Log is configured to *not* fsync() on all Append() calls
I20260929 03:17:59.212723 27830 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82e1ecef80f449029732e3ade3c4a092: No bootstrap required, opened a new log
I20260929 03:17:59.214145 27830 raft_consensus.cc:348] T 00000000000000000000000000000000 P 82e1ecef80f449029732e3ade3c4a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4b1269455e04063b0eb4e2aa46d0684" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 35553 } } peers { permanent_uuid: "82e1ecef80f449029732e3ade3c4a092" member_type: VOTER last_known_addr { host: "127.20.178.125" port: 35183 } }
I20260929 03:17:59.214262 27830 raft_consensus.cc:374] T 00000000000000000000000000000000 P 82e1ecef80f449029732e3ade3c4a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:17:59.214283 27830 raft_consensus.cc:729] T 00000000000000000000000000000000 P 82e1ecef80f449029732e3ade3c4a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82e1ecef80f449029732e3ade3c4a092, State: Initialized, Role: FOLLOWER
I20260929 03:17:59.214429 27830 consensus_queue.cc:260] T 00000000000000000000000000000000 P 82e1ecef80f449029732e3ade3c4a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4b1269455e04063b0eb4e2aa46d0684" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 35553 } } peers { permanent_uuid: "82e1ecef80f449029732e3ade3c4a092" member_type: VOTER last_known_addr { host: "127.20.178.125" port: 35183 } }
I20260929 03:17:59.214708 27830 sys_catalog.cc:564] T 00000000000000000000000000000000 P 82e1ecef80f449029732e3ade3c4a092 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:17:59.214670 27837 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82e1ecef80f449029732e3ade3c4a092 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4b1269455e04063b0eb4e2aa46d0684" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 35553 } } peers { permanent_uuid: "82e1ecef80f449029732e3ade3c4a092" member_type: VOTER last_known_addr { host: "127.20.178.125" port: 35183 } } }
I20260929 03:17:59.214777 27837 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82e1ecef80f449029732e3ade3c4a092 [sys.catalog]: This master's current role is: FOLLOWER
W20260929 03:17:59.216341 27848 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 82e1ecef80f449029732e3ade3c4a092: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 03:17:59.216393 27848 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 03:17:59.217936 27762 master_runner.cc:428] Detected that this master 82e1ecef80f449029732e3ade3c4a092 is joining an existing cluster
I20260929 03:17:59.217993 27762 master_runner.cc:432] Initiating AddMaster RPC to add 127.20.178.125:35183
I20260929 03:17:59.225908 27356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:59.218940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42442 (local address 127.20.178.126:35553)
0929 03:17:59.218983 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:59.218985 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:59.219128 (+ 143us) server_negotiation.cc:407] Connection header received
0929 03:17:59.219226 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:59.219227 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:59.219238 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:59.219300 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:17:59.219680 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:59.220429 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:59.221400 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:59.221607 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:59.221698 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:59.222848 (+ 1150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:17:59.222849 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:17:59.222849 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:17:59.222856 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:17:59.224449 (+ 1593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:59.224987 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:59.224988 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:59.224989 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:59.225063 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:59.225313 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:59.225314 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:59.225314 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:59.225446 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:17:59.225528 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 03:17:59.225726 (+ 198us) server_negotiation.cc:299] Negotiation successful
0929 03:17:59.225798 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260929 03:17:59.226106 27835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:59.219099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54150 (local address 127.20.178.125:35183)
0929 03:17:59.219130 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:59.219132 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:59.219141 (+ 9us) server_negotiation.cc:407] Connection header received
0929 03:17:59.219342 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:59.219342 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:59.219351 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:59.219403 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:17:59.219720 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:59.220196 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:59.221222 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:59.221542 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:59.221610 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:59.223013 (+ 1403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:17:59.223014 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:17:59.223015 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:17:59.223021 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:17:59.224896 (+ 1875us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:59.225251 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:59.225252 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:59.225253 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:59.225296 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:59.225477 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:59.225477 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:59.225478 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:59.225598 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:17:59.225666 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0929 03:17:59.225866 (+ 200us) server_negotiation.cc:299] Negotiation successful
0929 03:17:59.226002 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":54,"server-negotiator.queue_time_us":17}
I20260929 03:17:59.226137 27855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:59.218924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.125:35183 (local address 127.0.0.1:54150)
0929 03:17:59.219101 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0929 03:17:59.219110 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 03:17:59.219152 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:17:59.219419 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:17:59.219419 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:17:59.219586 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:17:59.219632 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:59.219634 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:59.220274 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:59.220275 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:59.221142 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:59.221143 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:59.221684 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:59.221686 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:59.221747 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:59.222281 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:17:59.222293 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:17:59.222910 (+ 617us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:17:59.224964 (+ 2054us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:17:59.224965 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:17:59.224965 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:17:59.225178 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:17:59.225354 (+ 176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:17:59.225354 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:17:59.225354 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:17:59.225408 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:17:59.225699 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:17:59.225700 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:17:59.225753 (+ 53us) client_negotiation.cc:769] Sending connection context
0929 03:17:59.225831 (+ 78us) client_negotiation.cc:240] Negotiation successful
0929 03:17:59.225933 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"mutex_wait_us":3,"spinlock_wait_cycles":8960,"thread_start_us":99,"threads_started":1}
I20260929 03:17:59.226688 27854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:59.218924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:35553 (local address 127.0.0.1:42442)
0929 03:17:59.219060 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0929 03:17:59.219071 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 03:17:59.219152 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:17:59.219300 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:17:59.219301 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:17:59.219511 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:17:59.219587 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:59.219589 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:59.220508 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:59.220509 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:59.221324 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:59.221326 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:59.221690 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:59.221691 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:59.221751 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:59.222358 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:17:59.222362 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:17:59.222742 (+ 380us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:17:59.224518 (+ 1776us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:17:59.224520 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:17:59.224520 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:17:59.224887 (+ 367us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:17:59.225170 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:17:59.225170 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:17:59.225171 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:17:59.225242 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:17:59.225519 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:17:59.225520 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:17:59.225559 (+ 39us) client_negotiation.cc:769] Sending connection context
0929 03:17:59.226521 (+ 962us) client_negotiation.cc:240] Negotiation successful
0929 03:17:59.226618 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":87,"threads_started":1}
Sep 29 03:17:59 dist-test-slave-c9qh krb5kdc[27253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651878, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.178.126@KRBTEST.COM for oryx/127.20.178.125@KRBTEST.COM
I20260929 03:17:59.235918 27835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:59.228909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54160 (local address 127.20.178.125:35183)
0929 03:17:59.228985 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:59.228987 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:59.229048 (+ 61us) server_negotiation.cc:407] Connection header received
0929 03:17:59.229145 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:59.229146 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:59.229153 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:59.229204 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:17:59.229670 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:59.230073 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:59.231174 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:59.231349 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:59.231406 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:59.233492 (+ 2086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:17:59.233493 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:17:59.233494 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:17:59.233500 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:17:59.234783 (+ 1283us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:59.235157 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:59.235158 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:59.235159 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:59.235220 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:59.235413 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:59.235414 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:59.235414 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:59.235510 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:17:59.235583 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 03:17:59.235760 (+ 177us) server_negotiation.cc:299] Negotiation successful
0929 03:17:59.235832 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 03:17:59.235952 27856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:59.228833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.125:35183 (local address 127.0.0.1:54160)
0929 03:17:59.228957 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0929 03:17:59.228998 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 03:17:59.229074 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:17:59.229209 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:17:59.229213 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:17:59.229532 (+ 319us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:17:59.229599 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:59.229601 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:59.230146 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:59.230147 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:59.231094 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:59.231096 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:59.231431 (+ 335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:59.231432 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:59.231510 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:59.231910 (+ 400us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:17:59.231915 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:17:59.233395 (+ 1480us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:17:59.234879 (+ 1484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:17:59.234886 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:17:59.234888 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:17:59.235084 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:17:59.235287 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:17:59.235288 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:17:59.235288 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:17:59.235351 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:17:59.235576 (+ 225us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:17:59.235582 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:17:59.235631 (+ 49us) client_negotiation.cc:769] Sending connection context
0929 03:17:59.235736 (+ 105us) client_negotiation.cc:240] Negotiation successful
0929 03:17:59.235844 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":74,"threads_started":1}
I20260929 03:17:59.236505 27298 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.20.178.125:35183: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "82e1ecef80f449029732e3ade3c4a092"
member_type: NON_VOTER
last_known_addr {
host: "127.20.178.125"
port: 35183
}
attrs {
promote: true
}
}
dest_uuid: "d4b1269455e04063b0eb4e2aa46d0684"
cas_config_opid_index: -1
I20260929 03:17:59.236850 27298 consensus_queue.cc:237] T 00000000000000000000000000000000 P d4b1269455e04063b0eb4e2aa46d0684 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d4b1269455e04063b0eb4e2aa46d0684" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 35553 } } peers { permanent_uuid: "82e1ecef80f449029732e3ade3c4a092" member_type: NON_VOTER last_known_addr { host: "127.20.178.125" port: 35183 } attrs { promote: true } }
I20260929 03:17:59.237344 27858 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d4b1269455e04063b0eb4e2aa46d0684 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 82e1ecef80f449029732e3ade3c4a092 (127.20.178.125) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d4b1269455e04063b0eb4e2aa46d0684" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 35553 } } peers { permanent_uuid: "82e1ecef80f449029732e3ade3c4a092" member_type: NON_VOTER last_known_addr { host: "127.20.178.125" port: 35183 } attrs { promote: true } } }
I20260929 03:17:59.237540 27804 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 82e1ecef80f449029732e3ade3c4a092 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:17:59.238328 27804 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 82e1ecef80f449029732e3ade3c4a092 [term 1 FOLLOWER]: Refusing update from remote peer d4b1269455e04063b0eb4e2aa46d0684: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260929 03:17:59.238358 27858 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.20.178.125:35183
I20260929 03:17:59.238572 27762 master.cc:537] Master@127.20.178.125:35183 shutting down...
I20260929 03:17:59.238744 27857 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4b1269455e04063b0eb4e2aa46d0684 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "d4b1269455e04063b0eb4e2aa46d0684" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d4b1269455e04063b0eb4e2aa46d0684" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 35553 } } peers { permanent_uuid: "82e1ecef80f449029732e3ade3c4a092" member_type: NON_VOTER last_known_addr { host: "127.20.178.125" port: 35183 } attrs { promote: true } } }
I20260929 03:17:59.238940 27857 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4b1269455e04063b0eb4e2aa46d0684 [sys.catalog]: This master's current role is: LEADER
I20260929 03:17:59.238759 27858 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d4b1269455e04063b0eb4e2aa46d0684 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82e1ecef80f449029732e3ade3c4a092" member_type: NON_VOTER last_known_addr { host: "127.20.178.125" port: 35183 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260929 03:17:59.239707 27282 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260929 03:17:59.240064 27282 consensus_peers.cc:487] T 00000000000000000000000000000000 P d4b1269455e04063b0eb4e2aa46d0684 -> Peer 82e1ecef80f449029732e3ade3c4a092 (127.20.178.125:35183): Couldn't send request to peer 82e1ecef80f449029732e3ade3c4a092. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260929 03:17:59.241051 27762 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 82e1ecef80f449029732e3ade3c4a092 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:17:59.241135 27762 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 82e1ecef80f449029732e3ade3c4a092 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:17:59.241185 27762 tablet_replica.cc:330] T 00000000000000000000000000000000 P 82e1ecef80f449029732e3ade3c4a092: stopping tablet replica
I20260929 03:17:59.253701 27762 master.cc:552] Master@127.20.178.125:35183 shutdown complete.
I20260929 03:17:59.253749 27762 master_runner.cc:305] Clearing existing system tablet
I20260929 03:17:59.253876 27762 fs_manager.cc:359] Metadata directory not provided
I20260929 03:17:59.253921 27762 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:17:59.254616 27762 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.001s
I20260929 03:17:59.255111 27863 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:17:59.255247 27762 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:17:59.255306 27762 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260929 03:17:59.255331 27762 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/minicluster-data/master-1/data,/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/minicluster-data/master-1/wal
uuid: "82e1ecef80f449029732e3ade3c4a092"
format_stamp: "Formatted at 2026-09-29 03:17:59 on dist-test-slave-c9qh"
server_key: "612b8ffb623b4a6b7a15f71234bdde3d"
server_key_iv: "444ce762e4607baaa494fcc4968d2523"
server_key_version: "encryptionkey@0"
I20260929 03:17:59.256064 27762 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 82e1ecef80f449029732e3ade3c4a092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:17:59.257781 27762 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 82e1ecef80f449029732e3ade3c4a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260929 03:17:59.257853 27762 log.cc:1196] T 00000000000000000000000000000000 P 82e1ecef80f449029732e3ade3c4a092: Deleting WAL directory at /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260929 03:17:59.258061 27762 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 82e1ecef80f449029732e3ade3c4a092: Deleting consensus metadata
I20260929 03:17:59.258180 27762 master_runner.cc:315] Copying system tablet from 127.20.178.126:35553
I20260929 03:17:59.258774 27762 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 82e1ecef80f449029732e3ade3c4a092: tablet copy: Beginning tablet copy session from remote peer at address 127.20.178.126:35553
I20260929 03:17:59.264739 27869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:59.259027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:35553 (local address 127.0.0.1:42452)
0929 03:17:59.259174 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0929 03:17:59.259192 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 03:17:59.259270 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:17:59.259415 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:17:59.259415 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:17:59.259569 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:17:59.259620 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:59.259622 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:59.260266 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:59.260267 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:59.260870 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:59.260872 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:59.261247 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:59.261248 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:59.261316 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:59.261778 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:17:59.261782 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:17:59.262047 (+ 265us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:17:59.263783 (+ 1736us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:17:59.263784 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:17:59.263785 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:17:59.263946 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:17:59.264141 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:17:59.264142 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:17:59.264142 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:17:59.264194 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:17:59.264447 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:17:59.264447 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:17:59.264480 (+ 33us) client_negotiation.cc:769] Sending connection context
0929 03:17:59.264529 (+ 49us) client_negotiation.cc:240] Negotiation successful
0929 03:17:59.264598 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":69,"threads_started":1}
I20260929 03:17:59.264781 27356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:59.259075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42452 (local address 127.20.178.126:35553)
0929 03:17:59.259119 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:59.259121 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:59.259255 (+ 134us) server_negotiation.cc:407] Connection header received
0929 03:17:59.259331 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:59.259332 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:59.259344 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:59.259399 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:17:59.259701 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:59.260187 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:59.260958 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:59.261163 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:59.261247 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:59.262154 (+ 907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:17:59.262156 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:17:59.262156 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:17:59.262163 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:17:59.263697 (+ 1534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:59.264013 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:59.264014 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:59.264015 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:59.264075 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:59.264255 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:59.264255 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:59.264256 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:59.264386 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:17:59.264454 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0929 03:17:59.264580 (+ 126us) server_negotiation.cc:299] Negotiation successful
0929 03:17:59.264690 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260929 03:17:59.265331 27318 tablet_copy_service.cc:140] P d4b1269455e04063b0eb4e2aa46d0684: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 82e1ecef80f449029732e3ade3c4a092 ({username='oryx', principal='oryx/127.20.178.125@KRBTEST.COM'} at 127.0.0.1:42452)
I20260929 03:17:59.265426 27318 tablet_copy_service.cc:161] P d4b1269455e04063b0eb4e2aa46d0684: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 82e1ecef80f449029732e3ade3c4a092 at {username='oryx', principal='oryx/127.20.178.125@KRBTEST.COM'} at 127.0.0.1:42452: session id = 82e1ecef80f449029732e3ade3c4a092-00000000000000000000000000000000
I20260929 03:17:59.265988 27318 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P d4b1269455e04063b0eb4e2aa46d0684: Tablet Copy: opened 0 blocks and 1 log segments
I20260929 03:17:59.266561 27762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260929 03:17:59.267365 27762 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 82e1ecef80f449029732e3ade3c4a092: tablet copy: Starting download of 0 data blocks...
I20260929 03:17:59.267515 27762 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 82e1ecef80f449029732e3ade3c4a092: tablet copy: Starting download of 1 WAL segments...
I20260929 03:17:59.268357 27762 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 82e1ecef80f449029732e3ade3c4a092: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260929 03:17:59.269126 27318 tablet_copy_service.cc:342] P d4b1269455e04063b0eb4e2aa46d0684: Request end of tablet copy session 82e1ecef80f449029732e3ade3c4a092-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.20.178.125@KRBTEST.COM'} at 127.0.0.1:42452
I20260929 03:17:59.269193 27318 tablet_copy_service.cc:433] P d4b1269455e04063b0eb4e2aa46d0684: ending tablet copy session 82e1ecef80f449029732e3ade3c4a092-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 82e1ecef80f449029732e3ade3c4a092
I20260929 03:17:59.271800 27762 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:17:59.272953 27871 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:17:59.273200 27870 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:17:59.273311 27873 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:17:59.274322 27762 server_base.cc:1034] running on GCE node
I20260929 03:17:59.274428 27762 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:17:59.274559 27762 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:17:59.275686 27762 hybrid_clock.cc:648] HybridClock initialized: now 1790651879275678 us; error 23 us; skew 500 ppm
Sep 29 03:17:59 dist-test-slave-c9qh krb5kdc[27253](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651879, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.178.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:17:59.276996 27762 init.cc:377] Logged in from keytab as oryx/127.20.178.125@KRBTEST.COM (short username oryx)
I20260929 03:17:59.277281 27762 webserver.cc:460] Webserver started at http://127.20.178.125:38233/ using document root <none> and password file <none>
I20260929 03:17:59.277397 27762 fs_manager.cc:359] Metadata directory not provided
I20260929 03:17:59.277451 27762 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:17:59.278025 27762 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260929 03:17:59.278512 27880 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:17:59.278652 27762 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260929 03:17:59.278704 27762 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/minicluster-data/master-1/data,/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/minicluster-data/master-1/wal
uuid: "82e1ecef80f449029732e3ade3c4a092"
format_stamp: "Formatted at 2026-09-29 03:17:59 on dist-test-slave-c9qh"
server_key: "612b8ffb623b4a6b7a15f71234bdde3d"
server_key_iv: "444ce762e4607baaa494fcc4968d2523"
server_key_version: "encryptionkey@0"
I20260929 03:17:59.278759 27762 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:17:59.286159 27762 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:17:59.287563 27762 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:17:59.290674 27762 rpc_server.cc:307] RPC server started. Bound to: 127.20.178.125:35183
I20260929 03:17:59.290967 27932 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.178.125:35183 every 8 connection(s)
I20260929 03:17:59.291162 27762 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/minicluster-data/master-1/data/info.pb
I20260929 03:17:59.291879 27933 sys_catalog.cc:263] Verifying existing consensus state
I20260929 03:17:59.292268 27933 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82e1ecef80f449029732e3ade3c4a092: Bootstrap starting.
I20260929 03:17:59.294490 27933 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82e1ecef80f449029732e3ade3c4a092: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 03:17:59.294716 27933 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82e1ecef80f449029732e3ade3c4a092: Bootstrap complete.
I20260929 03:17:59.294904 27933 raft_consensus.cc:348] T 00000000000000000000000000000000 P 82e1ecef80f449029732e3ade3c4a092 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d4b1269455e04063b0eb4e2aa46d0684" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 35553 } } peers { permanent_uuid: "82e1ecef80f449029732e3ade3c4a092" member_type: NON_VOTER last_known_addr { host: "127.20.178.125" port: 35183 } attrs { promote: true } }
I20260929 03:17:59.294966 27933 raft_consensus.cc:729] T 00000000000000000000000000000000 P 82e1ecef80f449029732e3ade3c4a092 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 82e1ecef80f449029732e3ade3c4a092, State: Initialized, Role: LEARNER
I20260929 03:17:59.294992 27933 consensus_queue.cc:260] T 00000000000000000000000000000000 P 82e1ecef80f449029732e3ade3c4a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d4b1269455e04063b0eb4e2aa46d0684" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 35553 } } peers { permanent_uuid: "82e1ecef80f449029732e3ade3c4a092" member_type: NON_VOTER last_known_addr { host: "127.20.178.125" port: 35183 } attrs { promote: true } }
I20260929 03:17:59.295171 27933 sys_catalog.cc:564] T 00000000000000000000000000000000 P 82e1ecef80f449029732e3ade3c4a092 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:17:59.295161 27936 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82e1ecef80f449029732e3ade3c4a092 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d4b1269455e04063b0eb4e2aa46d0684" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 35553 } } peers { permanent_uuid: "82e1ecef80f449029732e3ade3c4a092" member_type: NON_VOTER last_known_addr { host: "127.20.178.125" port: 35183 } attrs { promote: true } } }
I20260929 03:17:59.295265 27936 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82e1ecef80f449029732e3ade3c4a092 [sys.catalog]: This master's current role is: LEARNER
I20260929 03:17:59.296700 27947 catalog_manager.cc:1259] Loaded cluster ID: d4b12290c9754a30a1ed0ccfd1dc774e
I20260929 03:17:59.296769 27947 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 82e1ecef80f449029732e3ade3c4a092: loading cluster ID for follower catalog manager: success
I20260929 03:17:59.297896 27947 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 82e1ecef80f449029732e3ade3c4a092: acquiring CA information for follower catalog manager: success
I20260929 03:17:59.298157 27947 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 82e1ecef80f449029732e3ade3c4a092: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 03:17:59.315498 27948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:59.308343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54178 (local address 127.20.178.125:35183)
0929 03:17:59.308474 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:59.308476 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:59.308487 (+ 11us) server_negotiation.cc:407] Connection header received
0929 03:17:59.308512 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:59.308513 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:59.308524 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:59.308610 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:17:59.309162 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:59.310206 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:59.310910 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:59.311219 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:59.311296 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:59.312435 (+ 1139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:17:59.312436 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:17:59.312437 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:17:59.312444 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:17:59.314274 (+ 1830us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:59.314590 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:59.314591 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:59.314591 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:59.314653 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:59.314831 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:59.314832 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:59.314832 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:59.314961 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:17:59.315060 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0929 03:17:59.315144 (+ 84us) server_negotiation.cc:299] Negotiation successful
0929 03:17:59.315221 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":66,"threads_started":1}
WARNING: no policy specified for oryx/127.20.178.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.20.178.124@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.20.178.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/krb5kdc/oryx_127.20.178.124.keytab.
Entry for principal oryx/127.20.178.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/krb5kdc/oryx_127.20.178.124.keytab.
I20260929 03:17:59.339777 21193 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuoaHbX/build/release/bin/kudu
/tmp/dist-test-taskuoaHbX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.178.124:41439
--webserver_interface=127.20.178.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/krb5kdc/oryx_127.20.178.124.keytab
--principal=oryx/127.20.178.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.178.84:35629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.178.126:35553,127.20.178.125:35183,127.20.178.124:41439
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:17:59.419462 27951 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:17:59.419704 27951 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:17:59.419742 27951 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:17:59.420919 27951 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:17:59.420982 27951 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:17:59.421007 27951 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:17:59.421042 27951 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:17:59.421087 27951 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:17:59.422524 27951 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.178.84:35629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.20.178.126:35553,127.20.178.125:35183,127.20.178.124:41439
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/krb5kdc/oryx_127.20.178.124.keytab
--principal=oryx/127.20.178.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.178.124:41439
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.20.178.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27951
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:17:59.422789 27951 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:17:59.423003 27951 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:17:59.427950 27959 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:17:59.427968 27957 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:17:59.427990 27956 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:17:59.428068 27951 server_base.cc:1034] running on GCE node
I20260929 03:17:59.428332 27951 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:17:59.428607 27951 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:17:59.429781 27951 hybrid_clock.cc:648] HybridClock initialized: now 1790651879429772 us; error 29 us; skew 500 ppm
Sep 29 03:17:59 dist-test-slave-c9qh krb5kdc[27253](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651879, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.178.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:17:59.432261 27951 init.cc:377] Logged in from keytab as oryx/127.20.178.124@KRBTEST.COM (short username oryx)
I20260929 03:17:59.432680 27951 webserver.cc:460] Webserver started at http://127.20.178.124:40881/ using document root <none> and password file <none>
I20260929 03:17:59.432874 27951 fs_manager.cc:359] Metadata directory not provided
I20260929 03:17:59.432912 27951 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:17:59.432996 27951 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:17:59.433979 27951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/minicluster-data/master-2/data/instance:
uuid: "bce5b828f0a947c288a8cb13eb6c6819"
format_stamp: "Formatted at 2026-09-29 03:17:59 on dist-test-slave-c9qh"
server_key: "869608a61231f923333812312e848cc0"
server_key_iv: "058c06acb041fe2ff3f4d5a9e07ba83e"
server_key_version: "encryptionkey@0"
I20260929 03:17:59.434274 27951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/minicluster-data/master-2/wal/instance:
uuid: "bce5b828f0a947c288a8cb13eb6c6819"
format_stamp: "Formatted at 2026-09-29 03:17:59 on dist-test-slave-c9qh"
server_key: "869608a61231f923333812312e848cc0"
server_key_iv: "058c06acb041fe2ff3f4d5a9e07ba83e"
server_key_version: "encryptionkey@0"
I20260929 03:17:59.435451 27951 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 03:17:59.436072 27966 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:17:59.436249 27951 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 03:17:59.436333 27951 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/minicluster-data/master-2/data,/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/minicluster-data/master-2/wal
uuid: "bce5b828f0a947c288a8cb13eb6c6819"
format_stamp: "Formatted at 2026-09-29 03:17:59 on dist-test-slave-c9qh"
server_key: "869608a61231f923333812312e848cc0"
server_key_iv: "058c06acb041fe2ff3f4d5a9e07ba83e"
server_key_version: "encryptionkey@0"
I20260929 03:17:59.436405 27951 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:17:59.467648 27951 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:17:59.469713 27951 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:17:59.469866 27951 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:17:59.473629 27951 rpc_server.cc:307] RPC server started. Bound to: 127.20.178.124:41439
I20260929 03:17:59.473707 28018 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.178.124:41439 every 8 connection(s)
I20260929 03:17:59.474009 27951 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/minicluster-data/master-2/data/info.pb
I20260929 03:17:59.474700 21193 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuoaHbX/build/release/bin/kudu as pid 27951
I20260929 03:17:59.475160 28019 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:17:59.477116 28019 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.178.126" port: 35553 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 03:17:59 dist-test-slave-c9qh krb5kdc[27253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651878, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.20.178.124@KRBTEST.COM
Sep 29 03:17:59 dist-test-slave-c9qh krb5kdc[27253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651879, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.178.124@KRBTEST.COM for oryx/127.20.178.126@KRBTEST.COM
I20260929 03:17:59.486238 28021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:59.475323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44004 (local address 127.20.178.124:41439)
0929 03:17:59.475497 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:59.475505 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:59.475532 (+ 27us) server_negotiation.cc:407] Connection header received
0929 03:17:59.476196 (+ 664us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:59.476203 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:59.476264 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:59.476416 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:17:59.477135 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:59.478397 (+ 1262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:59.479454 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:59.479960 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:59.480108 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:59.482856 (+ 2748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:17:59.482862 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:17:59.482862 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:17:59.482868 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:17:59.484701 (+ 1833us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:59.485184 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:59.485186 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:59.485186 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:59.485270 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:59.485459 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:59.485459 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:59.485459 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:59.485665 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:17:59.485827 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0929 03:17:59.485949 (+ 122us) server_negotiation.cc:299] Negotiation successful
0929 03:17:59.486066 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":83,"threads_started":1}
I20260929 03:17:59.486384 27356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:59.477880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42464 (local address 127.20.178.126:35553)
0929 03:17:59.477925 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:59.477927 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:59.477965 (+ 38us) server_negotiation.cc:407] Connection header received
0929 03:17:59.478039 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:59.478039 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:59.478047 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:59.478109 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:17:59.478678 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:59.479419 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:59.480228 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:59.480522 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:59.480591 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:59.483607 (+ 3016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:17:59.483608 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:17:59.483608 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:17:59.483615 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:17:59.485201 (+ 1586us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:59.485611 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:59.485612 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:59.485612 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:59.485666 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:59.485876 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:59.485877 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:59.485877 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:59.485983 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:17:59.486056 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 03:17:59.486223 (+ 167us) server_negotiation.cc:299] Negotiation successful
0929 03:17:59.486297 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260929 03:17:59.486398 28022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:59.477715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:35553 (local address 127.0.0.1:42464)
0929 03:17:59.477873 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0929 03:17:59.477912 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 03:17:59.477981 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:17:59.478140 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:17:59.478143 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:17:59.478494 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:17:59.478578 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:59.478580 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:59.479506 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:59.479507 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:59.480143 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:59.480145 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:59.480699 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:59.480701 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:59.480784 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:59.481192 (+ 408us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:17:59.481202 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:17:59.483457 (+ 2255us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:17:59.485272 (+ 1815us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:17:59.485278 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:17:59.485280 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:17:59.485539 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:17:59.485740 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:17:59.485740 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:17:59.485741 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:17:59.485797 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:17:59.486047 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:17:59.486060 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:17:59.486111 (+ 51us) client_negotiation.cc:769] Sending connection context
0929 03:17:59.486195 (+ 84us) client_negotiation.cc:240] Negotiation successful
0929 03:17:59.486297 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":89,"threads_started":1}
I20260929 03:17:59.486934 28019 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.178.125" port: 35183 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 03:17:59 dist-test-slave-c9qh krb5kdc[27253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651879, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.178.124@KRBTEST.COM for oryx/127.20.178.125@KRBTEST.COM
I20260929 03:17:59.493144 28022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:59.487159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.125:35183 (local address 127.0.0.1:54194)
0929 03:17:59.487204 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0929 03:17:59.487214 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 03:17:59.487253 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:17:59.487396 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:17:59.487396 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:17:59.487521 (+ 125us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:17:59.487564 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:59.487565 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:59.488194 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:59.488195 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:59.488751 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:59.488752 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:59.489148 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:59.489149 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:59.489214 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:59.489520 (+ 306us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:17:59.489524 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:17:59.490580 (+ 1056us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:17:59.492153 (+ 1573us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:17:59.492154 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:17:59.492154 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:17:59.492326 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:17:59.492546 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:17:59.492547 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:17:59.492547 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:17:59.492604 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:17:59.492869 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:17:59.492870 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:17:59.492909 (+ 39us) client_negotiation.cc:769] Sending connection context
0929 03:17:59.492975 (+ 66us) client_negotiation.cc:240] Negotiation successful
0929 03:17:59.493062 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260929 03:17:59.493196 27948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:59.487250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54194 (local address 127.20.178.125:35183)
0929 03:17:59.487291 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:59.487294 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:59.487301 (+ 7us) server_negotiation.cc:407] Connection header received
0929 03:17:59.487325 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:59.487326 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:59.487336 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:59.487389 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:17:59.487647 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:59.488118 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:59.488841 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:59.489067 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:59.489148 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:59.490668 (+ 1520us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:17:59.490670 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:17:59.490670 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:17:59.490677 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:17:59.492055 (+ 1378us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:59.492412 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:59.492413 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:59.492414 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:59.492470 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:59.492693 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:59.492694 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:59.492695 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:59.492804 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:17:59.492878 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0929 03:17:59.493007 (+ 129us) server_negotiation.cc:299] Negotiation successful
0929 03:17:59.493082 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260929 03:17:59.493533 28019 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.178.124" port: 41439 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 03:17:59 dist-test-slave-c9qh krb5kdc[27253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651879, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.178.124@KRBTEST.COM for oryx/127.20.178.124@KRBTEST.COM
I20260929 03:17:59.499191 28022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:59.493760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.124:41439 (local address 127.0.0.1:44014)
0929 03:17:59.493805 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0929 03:17:59.493815 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 03:17:59.493849 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:17:59.493946 (+ 97us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:17:59.493947 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:17:59.494044 (+ 97us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:17:59.494067 (+ 23us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:59.494068 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:59.494606 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:59.494607 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:59.495137 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:59.495138 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:59.495483 (+ 345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:59.495484 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:59.495531 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:59.495865 (+ 334us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:17:59.495868 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:17:59.497186 (+ 1318us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:17:59.498383 (+ 1197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:17:59.498384 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:17:59.498384 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:17:59.498557 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:17:59.498736 (+ 179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:17:59.498737 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:17:59.498737 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:17:59.498785 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:17:59.498974 (+ 189us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:17:59.498975 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:17:59.499007 (+ 32us) client_negotiation.cc:769] Sending connection context
0929 03:17:59.499059 (+ 52us) client_negotiation.cc:240] Negotiation successful
0929 03:17:59.499125 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260929 03:17:59.499248 28021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:59.493805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44014 (local address 127.20.178.124:41439)
0929 03:17:59.493857 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:59.493859 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:59.493867 (+ 8us) server_negotiation.cc:407] Connection header received
0929 03:17:59.493884 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:59.493884 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:59.493892 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:59.493942 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:17:59.494120 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:59.494544 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:59.495200 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:59.495380 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:59.495457 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:59.497270 (+ 1813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:17:59.497270 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:17:59.497271 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:17:59.497276 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:17:59.498315 (+ 1039us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:59.498619 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:59.498620 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:59.498620 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:59.498664 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:59.498833 (+ 169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:59.498833 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:59.498833 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:59.498916 (+ 83us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:17:59.498963 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0929 03:17:59.499074 (+ 111us) server_negotiation.cc:299] Negotiation successful
0929 03:17:59.499166 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":39,"server-negotiator.queue_time_us":37}
I20260929 03:17:59.500535 28019 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bce5b828f0a947c288a8cb13eb6c6819: Bootstrap starting.
I20260929 03:17:59.501338 28019 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bce5b828f0a947c288a8cb13eb6c6819: Neither blocks nor log segments found. Creating new log.
I20260929 03:17:59.501555 28019 log.cc:824] T 00000000000000000000000000000000 P bce5b828f0a947c288a8cb13eb6c6819: Log is configured to *not* fsync() on all Append() calls
I20260929 03:17:59.502202 28019 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bce5b828f0a947c288a8cb13eb6c6819: No bootstrap required, opened a new log
I20260929 03:17:59.503472 28019 raft_consensus.cc:348] T 00000000000000000000000000000000 P bce5b828f0a947c288a8cb13eb6c6819 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4b1269455e04063b0eb4e2aa46d0684" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 35553 } } peers { permanent_uuid: "82e1ecef80f449029732e3ade3c4a092" member_type: VOTER last_known_addr { host: "127.20.178.125" port: 35183 } } peers { permanent_uuid: "bce5b828f0a947c288a8cb13eb6c6819" member_type: VOTER last_known_addr { host: "127.20.178.124" port: 41439 } }
I20260929 03:17:59.503582 28019 raft_consensus.cc:374] T 00000000000000000000000000000000 P bce5b828f0a947c288a8cb13eb6c6819 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:17:59.503605 28019 raft_consensus.cc:729] T 00000000000000000000000000000000 P bce5b828f0a947c288a8cb13eb6c6819 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bce5b828f0a947c288a8cb13eb6c6819, State: Initialized, Role: FOLLOWER
I20260929 03:17:59.503721 28019 consensus_queue.cc:260] T 00000000000000000000000000000000 P bce5b828f0a947c288a8cb13eb6c6819 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4b1269455e04063b0eb4e2aa46d0684" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 35553 } } peers { permanent_uuid: "82e1ecef80f449029732e3ade3c4a092" member_type: VOTER last_known_addr { host: "127.20.178.125" port: 35183 } } peers { permanent_uuid: "bce5b828f0a947c288a8cb13eb6c6819" member_type: VOTER last_known_addr { host: "127.20.178.124" port: 41439 } }
I20260929 03:17:59.503966 28024 sys_catalog.cc:455] T 00000000000000000000000000000000 P bce5b828f0a947c288a8cb13eb6c6819 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4b1269455e04063b0eb4e2aa46d0684" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 35553 } } peers { permanent_uuid: "82e1ecef80f449029732e3ade3c4a092" member_type: VOTER last_known_addr { host: "127.20.178.125" port: 35183 } } peers { permanent_uuid: "bce5b828f0a947c288a8cb13eb6c6819" member_type: VOTER last_known_addr { host: "127.20.178.124" port: 41439 } } }
I20260929 03:17:59.504036 28019 sys_catalog.cc:564] T 00000000000000000000000000000000 P bce5b828f0a947c288a8cb13eb6c6819 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:17:59.504050 28024 sys_catalog.cc:458] T 00000000000000000000000000000000 P bce5b828f0a947c288a8cb13eb6c6819 [sys.catalog]: This master's current role is: FOLLOWER
W20260929 03:17:59.505726 28035 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bce5b828f0a947c288a8cb13eb6c6819: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 03:17:59.505802 28035 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 03:17:59.507740 27951 master_runner.cc:428] Detected that this master bce5b828f0a947c288a8cb13eb6c6819 is joining an existing cluster
I20260929 03:17:59.507787 27951 master_runner.cc:432] Initiating AddMaster RPC to add 127.20.178.124:41439
I20260929 03:17:59.532439 27948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:59.508846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54196 (local address 127.20.178.125:35183)
0929 03:17:59.508892 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:59.508894 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:59.509064 (+ 170us) server_negotiation.cc:407] Connection header received
0929 03:17:59.509167 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:59.509167 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:59.509177 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:59.509206 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:17:59.509670 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:59.510422 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:59.511490 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:59.511798 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:59.511840 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:59.529542 (+ 17702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:17:59.529544 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:17:59.529544 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:17:59.529552 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:17:59.531157 (+ 1605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:59.531625 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:59.531626 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:59.531627 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:59.531689 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:59.531888 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:59.531889 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:59.531890 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:59.532019 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:17:59.532061 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0929 03:17:59.532260 (+ 199us) server_negotiation.cc:299] Negotiation successful
0929 03:17:59.532342 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260929 03:17:59.532850 28021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:59.509045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44020 (local address 127.20.178.124:41439)
0929 03:17:59.509078 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:59.509080 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:59.509356 (+ 276us) server_negotiation.cc:407] Connection header received
0929 03:17:59.509434 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:59.509435 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:59.509443 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:59.509469 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:17:59.509862 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:59.510574 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:59.511698 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:59.511959 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:59.511994 (+ 35us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:59.529622 (+ 17628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:17:59.529624 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:17:59.529624 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:17:59.529630 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:17:59.531515 (+ 1885us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:59.531906 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:59.531908 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:59.531908 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:59.531961 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:59.532143 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:59.532144 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:59.532144 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:59.532278 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:17:59.532319 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0929 03:17:59.532497 (+ 178us) server_negotiation.cc:299] Negotiation successful
0929 03:17:59.532590 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260929 03:17:59.532857 28042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:59.508771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.125:35183 (local address 127.0.0.1:54196)
0929 03:17:59.508995 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0929 03:17:59.509007 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 03:17:59.509092 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:17:59.509267 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:17:59.509268 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:17:59.509507 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:17:59.509582 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:59.509584 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:59.510517 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:59.510518 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:59.511391 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:59.511394 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:59.511927 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:59.511928 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:59.511987 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:59.512403 (+ 416us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:17:59.512407 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:17:59.529418 (+ 17011us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:17:59.531251 (+ 1833us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:17:59.531252 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:17:59.531252 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:17:59.531543 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:17:59.531762 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:17:59.531762 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:17:59.531763 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:17:59.531820 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:17:59.532108 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:17:59.532108 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:17:59.532152 (+ 44us) client_negotiation.cc:769] Sending connection context
0929 03:17:59.532573 (+ 421us) client_negotiation.cc:240] Negotiation successful
0929 03:17:59.532733 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":92,"threads_started":1}
I20260929 03:17:59.533842 28043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:59.509015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.124:41439 (local address 127.0.0.1:44020)
0929 03:17:59.509311 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0929 03:17:59.509319 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0929 03:17:59.509374 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:17:59.509517 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:17:59.509518 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:17:59.509732 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:17:59.509788 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:59.509790 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:59.510663 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:59.510664 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:59.511608 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:59.511610 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:59.512060 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:59.512061 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:59.512391 (+ 330us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:59.512801 (+ 410us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:17:59.512806 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:17:59.529521 (+ 16715us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:17:59.531605 (+ 2084us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:17:59.531606 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:17:59.531606 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:17:59.531824 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:17:59.532028 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:17:59.532028 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:17:59.532029 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:17:59.532081 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:17:59.532360 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:17:59.532361 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:17:59.532398 (+ 37us) client_negotiation.cc:769] Sending connection context
0929 03:17:59.533203 (+ 805us) client_negotiation.cc:240] Negotiation successful
0929 03:17:59.533566 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"mutex_wait_us":66,"thread_start_us":63,"threads_started":1}
I20260929 03:17:59.534665 28041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:59.508877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:35553 (local address 127.0.0.1:42476)
0929 03:17:59.509128 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0929 03:17:59.509137 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 03:17:59.509221 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:17:59.509393 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:17:59.509394 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:17:59.509697 (+ 303us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:17:59.509757 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:59.509760 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:59.510674 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:59.510674 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:59.511608 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:59.511610 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:59.512109 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:59.512110 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:59.512176 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:59.529183 (+ 17007us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:17:59.529189 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:17:59.530214 (+ 1025us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:17:59.532613 (+ 2399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:17:59.532613 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:17:59.532614 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:17:59.533743 (+ 1129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:17:59.533971 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:17:59.533972 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:17:59.533972 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:17:59.534030 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:17:59.534337 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:17:59.534338 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:17:59.534377 (+ 39us) client_negotiation.cc:769] Sending connection context
0929 03:17:59.534466 (+ 89us) client_negotiation.cc:240] Negotiation successful
0929 03:17:59.534565 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"mutex_wait_us":42,"thread_start_us":60,"threads_started":1}
I20260929 03:17:59.534675 27356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:59.508804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42476 (local address 127.20.178.126:35553)
0929 03:17:59.508857 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:59.508859 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:59.509202 (+ 343us) server_negotiation.cc:407] Connection header received
0929 03:17:59.509281 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:59.509282 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:59.509294 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:59.509348 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:17:59.509853 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:59.510574 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:59.511714 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:59.512006 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:59.512044 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:59.530770 (+ 18726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:17:59.530771 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:17:59.530772 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:17:59.530781 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:17:59.532514 (+ 1733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:59.533833 (+ 1319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:59.533834 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:59.533834 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:59.533895 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:59.534098 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:59.534100 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:59.534100 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:59.534248 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:17:59.534312 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0929 03:17:59.534504 (+ 192us) server_negotiation.cc:299] Negotiation successful
0929 03:17:59.534577 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260929 03:17:59.537429 21193 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuoaHbX/build/release/bin/kudu with pid 27360
Sep 29 03:17:59 dist-test-slave-c9qh krb5kdc[27253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651878, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.178.126@KRBTEST.COM for oryx/127.20.178.124@KRBTEST.COM
I20260929 03:17:59.544231 27856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:59.536534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.124:41439 (local address 127.0.0.1:44034)
0929 03:17:59.536584 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0929 03:17:59.536594 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 03:17:59.536648 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:17:59.536777 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:17:59.536778 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:17:59.536921 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:17:59.536956 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:59.536957 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:59.537937 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:59.537938 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:59.538849 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:59.538852 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:59.539315 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:59.539316 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:59.539394 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:59.539844 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:17:59.539849 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:17:59.541458 (+ 1609us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:17:59.543221 (+ 1763us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:17:59.543222 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:17:59.543223 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:17:59.543429 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:17:59.543616 (+ 187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:17:59.543617 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:17:59.543617 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:17:59.543669 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:17:59.543936 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:17:59.543936 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:17:59.543977 (+ 41us) client_negotiation.cc:769] Sending connection context
0929 03:17:59.544036 (+ 59us) client_negotiation.cc:240] Negotiation successful
0929 03:17:59.544135 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260929 03:17:59.544554 21193 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuoaHbX/build/release/bin/kudu
/tmp/dist-test-taskuoaHbX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.178.65:45139
--local_ip_for_outbound_sockets=127.20.178.65
--tserver_master_addrs=127.20.178.126:35553
--webserver_port=36929
--webserver_interface=127.20.178.65
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/krb5kdc/oryx_127.20.178.65.keytab
--principal=oryx/127.20.178.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.178.84:35629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.20.178.126:35553,127.20.178.125:35183
--tserver_master_addrs=127.20.178.126:35553,127.20.178.125:35183,127.20.178.124:41439 with env {KRB5CCNAME=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 03:17:59.544873 28021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:59.536592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44034 (local address 127.20.178.124:41439)
0929 03:17:59.536649 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:59.536681 (+ 32us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:59.536689 (+ 8us) server_negotiation.cc:407] Connection header received
0929 03:17:59.536718 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:59.536718 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:59.536727 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:59.536774 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:17:59.537037 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:59.537691 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:59.538929 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:59.539230 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:59.539304 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:59.541547 (+ 2243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:17:59.541548 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:17:59.541548 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:17:59.541555 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:17:59.543128 (+ 1573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:59.543502 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:59.543503 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:59.543503 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:59.543552 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:59.543745 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:59.543745 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:59.543745 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:59.543858 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:17:59.543912 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0929 03:17:59.544563 (+ 651us) server_negotiation.cc:299] Negotiation successful
0929 03:17:59.544787 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260929 03:17:59.545271 27298 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.20.178.124:41439: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "bce5b828f0a947c288a8cb13eb6c6819"
member_type: NON_VOTER
last_known_addr {
host: "127.20.178.124"
port: 41439
}
attrs {
promote: true
}
}
dest_uuid: "d4b1269455e04063b0eb4e2aa46d0684"
cas_config_opid_index: 5
I20260929 03:17:59.545511 27298 consensus_queue.cc:237] T 00000000000000000000000000000000 P d4b1269455e04063b0eb4e2aa46d0684 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d4b1269455e04063b0eb4e2aa46d0684" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 35553 } } peers { permanent_uuid: "82e1ecef80f449029732e3ade3c4a092" member_type: NON_VOTER last_known_addr { host: "127.20.178.125" port: 35183 } attrs { promote: true } } peers { permanent_uuid: "bce5b828f0a947c288a8cb13eb6c6819" member_type: NON_VOTER last_known_addr { host: "127.20.178.124" port: 41439 } attrs { promote: true } }
I20260929 03:17:59.546027 27859 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d4b1269455e04063b0eb4e2aa46d0684 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER bce5b828f0a947c288a8cb13eb6c6819 (127.20.178.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d4b1269455e04063b0eb4e2aa46d0684" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 35553 } } peers { permanent_uuid: "82e1ecef80f449029732e3ade3c4a092" member_type: NON_VOTER last_known_addr { host: "127.20.178.125" port: 35183 } attrs { promote: true } } peers { permanent_uuid: "bce5b828f0a947c288a8cb13eb6c6819" member_type: NON_VOTER last_known_addr { host: "127.20.178.124" port: 41439 } attrs { promote: true } } }
I20260929 03:17:59.547108 27859 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.20.178.124:41439
I20260929 03:17:59.547179 27859 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4b1269455e04063b0eb4e2aa46d0684 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "d4b1269455e04063b0eb4e2aa46d0684" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d4b1269455e04063b0eb4e2aa46d0684" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 35553 } } peers { permanent_uuid: "82e1ecef80f449029732e3ade3c4a092" member_type: NON_VOTER last_known_addr { host: "127.20.178.125" port: 35183 } attrs { promote: true } } peers { permanent_uuid: "bce5b828f0a947c288a8cb13eb6c6819" member_type: NON_VOTER last_known_addr { host: "127.20.178.124" port: 41439 } attrs { promote: true } } }
I20260929 03:17:59.547228 27859 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4b1269455e04063b0eb4e2aa46d0684 [sys.catalog]: This master's current role is: LEADER
I20260929 03:17:59.547667 27993 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bce5b828f0a947c288a8cb13eb6c6819 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:17:59.548666 27993 raft_consensus.cc:1261] T 00000000000000000000000000000000 P bce5b828f0a947c288a8cb13eb6c6819 [term 1 FOLLOWER]: Refusing update from remote peer d4b1269455e04063b0eb4e2aa46d0684: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260929 03:17:59.548869 27951 master.cc:537] Master@127.20.178.124:41439 shutting down...
I20260929 03:17:59.548926 27858 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d4b1269455e04063b0eb4e2aa46d0684 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bce5b828f0a947c288a8cb13eb6c6819" member_type: NON_VOTER last_known_addr { host: "127.20.178.124" port: 41439 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260929 03:17:59.549588 27285 consensus_peers.cc:487] T 00000000000000000000000000000000 P d4b1269455e04063b0eb4e2aa46d0684 -> Peer bce5b828f0a947c288a8cb13eb6c6819 (127.20.178.124:41439): Couldn't send request to peer bce5b828f0a947c288a8cb13eb6c6819. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260929 03:17:59.551396 27856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:59.545975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.125:35183 (local address 127.0.0.1:54210)
0929 03:17:59.547605 (+ 1630us) negotiation.cc:107] Waiting for socket to connect
0929 03:17:59.547616 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 03:17:59.547661 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:17:59.547903 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:17:59.547904 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:17:59.547906 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:17:59.547955 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:59.547956 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:59.548889 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:59.548889 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:59.550226 (+ 1337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:59.550230 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:59.551149 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:59.551150 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:59.551208 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:59.551225 (+ 17us) client_negotiation.cc:769] Sending connection context
0929 03:17:59.551278 (+ 53us) client_negotiation.cc:240] Negotiation successful
0929 03:17:59.551318 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1613}
I20260929 03:17:59.551797 27948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:59.546038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54210 (local address 127.20.178.125:35183)
0929 03:17:59.546076 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:59.546078 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:59.547731 (+ 1653us) server_negotiation.cc:407] Connection header received
0929 03:17:59.547760 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:59.547761 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:59.547785 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:59.547844 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:17:59.548033 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:59.548795 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:59.550326 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:59.550971 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:59.551042 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:59.551115 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 03:17:59.551652 (+ 537us) server_negotiation.cc:299] Negotiation successful
0929 03:17:59.551713 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260929 03:17:59.551826 27951 raft_consensus.cc:2229] T 00000000000000000000000000000000 P bce5b828f0a947c288a8cb13eb6c6819 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:17:59.551901 27951 raft_consensus.cc:2258] T 00000000000000000000000000000000 P bce5b828f0a947c288a8cb13eb6c6819 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:17:59.551954 27951 tablet_replica.cc:330] T 00000000000000000000000000000000 P bce5b828f0a947c288a8cb13eb6c6819: stopping tablet replica
I20260929 03:17:59.552052 27907 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 82e1ecef80f449029732e3ade3c4a092 [term 1 LEARNER]: Refusing update from remote peer d4b1269455e04063b0eb4e2aa46d0684: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260929 03:17:59.552266 27858 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d4b1269455e04063b0eb4e2aa46d0684 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82e1ecef80f449029732e3ade3c4a092" member_type: NON_VOTER last_known_addr { host: "127.20.178.125" port: 35183 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260929 03:17:59.552834 27907 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 82e1ecef80f449029732e3ade3c4a092 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER bce5b828f0a947c288a8cb13eb6c6819 (127.20.178.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d4b1269455e04063b0eb4e2aa46d0684" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 35553 } } peers { permanent_uuid: "82e1ecef80f449029732e3ade3c4a092" member_type: NON_VOTER last_known_addr { host: "127.20.178.125" port: 35183 } attrs { promote: true } } peers { permanent_uuid: "bce5b828f0a947c288a8cb13eb6c6819" member_type: NON_VOTER last_known_addr { host: "127.20.178.124" port: 41439 } attrs { promote: true } } }
I20260929 03:17:59.554013 27936 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82e1ecef80f449029732e3ade3c4a092 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d4b1269455e04063b0eb4e2aa46d0684. Latest consensus state: current_term: 1 leader_uuid: "d4b1269455e04063b0eb4e2aa46d0684" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d4b1269455e04063b0eb4e2aa46d0684" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 35553 } } peers { permanent_uuid: "82e1ecef80f449029732e3ade3c4a092" member_type: NON_VOTER last_known_addr { host: "127.20.178.125" port: 35183 } attrs { promote: true } } peers { permanent_uuid: "bce5b828f0a947c288a8cb13eb6c6819" member_type: NON_VOTER last_known_addr { host: "127.20.178.124" port: 41439 } attrs { promote: true } } }
I20260929 03:17:59.554097 27936 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82e1ecef80f449029732e3ade3c4a092 [sys.catalog]: This master's current role is: LEARNER
I20260929 03:17:59.554139 27936 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82e1ecef80f449029732e3ade3c4a092 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d4b1269455e04063b0eb4e2aa46d0684" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d4b1269455e04063b0eb4e2aa46d0684" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 35553 } } peers { permanent_uuid: "82e1ecef80f449029732e3ade3c4a092" member_type: NON_VOTER last_known_addr { host: "127.20.178.125" port: 35183 } attrs { promote: true } } peers { permanent_uuid: "bce5b828f0a947c288a8cb13eb6c6819" member_type: NON_VOTER last_known_addr { host: "127.20.178.124" port: 41439 } attrs { promote: true } } }
I20260929 03:17:59.554194 27936 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82e1ecef80f449029732e3ade3c4a092 [sys.catalog]: This master's current role is: LEARNER
I20260929 03:17:59.555356 27951 master.cc:552] Master@127.20.178.124:41439 shutdown complete.
I20260929 03:17:59.555421 27951 master_runner.cc:305] Clearing existing system tablet
I20260929 03:17:59.555568 27951 fs_manager.cc:359] Metadata directory not provided
I20260929 03:17:59.555625 27951 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:17:59.556147 27858 raft_consensus.cc:1050] T 00000000000000000000000000000000 P d4b1269455e04063b0eb4e2aa46d0684: attempting to promote NON_VOTER 82e1ecef80f449029732e3ade3c4a092 to VOTER
I20260929 03:17:59.556348 27858 consensus_queue.cc:237] T 00000000000000000000000000000000 P d4b1269455e04063b0eb4e2aa46d0684 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d4b1269455e04063b0eb4e2aa46d0684" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 35553 } } peers { permanent_uuid: "82e1ecef80f449029732e3ade3c4a092" member_type: VOTER last_known_addr { host: "127.20.178.125" port: 35183 } attrs { promote: false } } peers { permanent_uuid: "bce5b828f0a947c288a8cb13eb6c6819" member_type: NON_VOTER last_known_addr { host: "127.20.178.124" port: 41439 } attrs { promote: true } }
I20260929 03:17:59.556768 27907 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 82e1ecef80f449029732e3ade3c4a092 [term 1 LEARNER]: Refusing update from remote peer d4b1269455e04063b0eb4e2aa46d0684: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260929 03:17:59.556877 27856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:59.556775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.124:41439 (local address 127.0.0.1:44038)
0929 03:17:59.556814 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0929 03:17:59.556853 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.178.124:41439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260929 03:17:59.556984 27859 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d4b1269455e04063b0eb4e2aa46d0684 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82e1ecef80f449029732e3ade3c4a092" member_type: VOTER last_known_addr { host: "127.20.178.125" port: 35183 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260929 03:17:59.557150 27856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:59.557076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.124:41439 (local address 127.0.0.1:44046)
0929 03:17:59.557109 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0929 03:17:59.557131 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.178.124:41439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
W20260929 03:17:59.557215 27285 consensus_peers.cc:487] T 00000000000000000000000000000000 P d4b1269455e04063b0eb4e2aa46d0684 -> Peer bce5b828f0a947c288a8cb13eb6c6819 (127.20.178.124:41439): Couldn't send request to peer bce5b828f0a947c288a8cb13eb6c6819. Status: Network error: Client connection negotiation failed: client connection to 127.20.178.124:41439: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260929 03:17:59.557298 27951 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 03:17:59.557765 27859 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d4b1269455e04063b0eb4e2aa46d0684 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 82e1ecef80f449029732e3ade3c4a092 (127.20.178.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d4b1269455e04063b0eb4e2aa46d0684" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 35553 } } peers { permanent_uuid: "82e1ecef80f449029732e3ade3c4a092" member_type: VOTER last_known_addr { host: "127.20.178.125" port: 35183 } attrs { promote: false } } peers { permanent_uuid: "bce5b828f0a947c288a8cb13eb6c6819" member_type: NON_VOTER last_known_addr { host: "127.20.178.124" port: 41439 } attrs { promote: true } } }
I20260929 03:17:59.557901 27907 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 82e1ecef80f449029732e3ade3c4a092 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 82e1ecef80f449029732e3ade3c4a092 (127.20.178.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d4b1269455e04063b0eb4e2aa46d0684" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 35553 } } peers { permanent_uuid: "82e1ecef80f449029732e3ade3c4a092" member_type: VOTER last_known_addr { host: "127.20.178.125" port: 35183 } attrs { promote: false } } peers { permanent_uuid: "bce5b828f0a947c288a8cb13eb6c6819" member_type: NON_VOTER last_known_addr { host: "127.20.178.124" port: 41439 } attrs { promote: true } } }
I20260929 03:17:59.558621 27859 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4b1269455e04063b0eb4e2aa46d0684 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "d4b1269455e04063b0eb4e2aa46d0684" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d4b1269455e04063b0eb4e2aa46d0684" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 35553 } } peers { permanent_uuid: "82e1ecef80f449029732e3ade3c4a092" member_type: VOTER last_known_addr { host: "127.20.178.125" port: 35183 } attrs { promote: false } } peers { permanent_uuid: "bce5b828f0a947c288a8cb13eb6c6819" member_type: NON_VOTER last_known_addr { host: "127.20.178.124" port: 41439 } attrs { promote: true } } }
I20260929 03:17:59.558657 27936 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82e1ecef80f449029732e3ade3c4a092 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d4b1269455e04063b0eb4e2aa46d0684" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d4b1269455e04063b0eb4e2aa46d0684" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 35553 } } peers { permanent_uuid: "82e1ecef80f449029732e3ade3c4a092" member_type: VOTER last_known_addr { host: "127.20.178.125" port: 35183 } attrs { promote: false } } peers { permanent_uuid: "bce5b828f0a947c288a8cb13eb6c6819" member_type: NON_VOTER last_known_addr { host: "127.20.178.124" port: 41439 } attrs { promote: true } } }
I20260929 03:17:59.558709 27859 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4b1269455e04063b0eb4e2aa46d0684 [sys.catalog]: This master's current role is: LEADER
I20260929 03:17:59.558718 27936 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82e1ecef80f449029732e3ade3c4a092 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 03:17:59.558766 27859 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4b1269455e04063b0eb4e2aa46d0684 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d4b1269455e04063b0eb4e2aa46d0684" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d4b1269455e04063b0eb4e2aa46d0684" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 35553 } } peers { permanent_uuid: "82e1ecef80f449029732e3ade3c4a092" member_type: VOTER last_known_addr { host: "127.20.178.125" port: 35183 } attrs { promote: false } } peers { permanent_uuid: "bce5b828f0a947c288a8cb13eb6c6819" member_type: NON_VOTER last_known_addr { host: "127.20.178.124" port: 41439 } attrs { promote: true } } }
I20260929 03:17:59.558823 27859 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4b1269455e04063b0eb4e2aa46d0684 [sys.catalog]: This master's current role is: LEADER
I20260929 03:17:59.559106 28048 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:17:59.559233 27951 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:17:59.559280 27951 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.000s
I20260929 03:17:59.559309 27951 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/minicluster-data/master-2/data,/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/minicluster-data/master-2/wal
uuid: "bce5b828f0a947c288a8cb13eb6c6819"
format_stamp: "Formatted at 2026-09-29 03:17:59 on dist-test-slave-c9qh"
server_key: "869608a61231f923333812312e848cc0"
server_key_iv: "058c06acb041fe2ff3f4d5a9e07ba83e"
server_key_version: "encryptionkey@0"
I20260929 03:17:59.559911 27951 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P bce5b828f0a947c288a8cb13eb6c6819: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:17:59.561096 27951 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P bce5b828f0a947c288a8cb13eb6c6819: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260929 03:17:59.561188 27951 log.cc:1196] T 00000000000000000000000000000000 P bce5b828f0a947c288a8cb13eb6c6819: Deleting WAL directory at /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260929 03:17:59.561393 27951 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P bce5b828f0a947c288a8cb13eb6c6819: Deleting consensus metadata
I20260929 03:17:59.563723 27951 master_runner.cc:315] Copying system tablet from 127.20.178.126:35553
I20260929 03:17:59.564405 27951 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P bce5b828f0a947c288a8cb13eb6c6819: tablet copy: Beginning tablet copy session from remote peer at address 127.20.178.126:35553
I20260929 03:17:59.570519 27356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:59.564757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42482 (local address 127.20.178.126:35553)
0929 03:17:59.564800 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:59.564803 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:59.564921 (+ 118us) server_negotiation.cc:407] Connection header received
0929 03:17:59.564979 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:59.564980 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:59.564990 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:59.565058 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:17:59.565388 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:59.565871 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:59.566574 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:59.566776 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:59.566862 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:59.567817 (+ 955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:17:59.567819 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:17:59.567819 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:17:59.567825 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:17:59.569374 (+ 1549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:59.569727 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:59.569728 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:59.569728 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:59.569788 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:59.570014 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:59.570014 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:59.570015 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:59.570129 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:17:59.570205 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0929 03:17:59.570359 (+ 154us) server_negotiation.cc:299] Negotiation successful
0929 03:17:59.570433 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260929 03:17:59.570538 28054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:59.564736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:35553 (local address 127.0.0.1:42482)
0929 03:17:59.564845 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0929 03:17:59.564854 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 03:17:59.564917 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:17:59.565077 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:17:59.565078 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:17:59.565260 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:17:59.565311 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:59.565312 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:59.565954 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:59.565955 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:59.566489 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:59.566491 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:59.566864 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:59.566865 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:59.566929 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:59.567396 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:17:59.567401 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:17:59.567709 (+ 308us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:17:59.569476 (+ 1767us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:17:59.569477 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:17:59.569477 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:17:59.569651 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:17:59.569860 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:17:59.569861 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:17:59.569861 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:17:59.569946 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:17:59.570202 (+ 256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:17:59.570203 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:17:59.570248 (+ 45us) client_negotiation.cc:769] Sending connection context
0929 03:17:59.570335 (+ 87us) client_negotiation.cc:240] Negotiation successful
0929 03:17:59.570433 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":62,"threads_started":1}
I20260929 03:17:59.571084 27318 tablet_copy_service.cc:140] P d4b1269455e04063b0eb4e2aa46d0684: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer bce5b828f0a947c288a8cb13eb6c6819 ({username='oryx', principal='oryx/127.20.178.124@KRBTEST.COM'} at 127.0.0.1:42482)
I20260929 03:17:59.571159 27318 tablet_copy_service.cc:161] P d4b1269455e04063b0eb4e2aa46d0684: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer bce5b828f0a947c288a8cb13eb6c6819 at {username='oryx', principal='oryx/127.20.178.124@KRBTEST.COM'} at 127.0.0.1:42482: session id = bce5b828f0a947c288a8cb13eb6c6819-00000000000000000000000000000000
I20260929 03:17:59.571597 27318 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P d4b1269455e04063b0eb4e2aa46d0684: Tablet Copy: opened 0 blocks and 1 log segments
I20260929 03:17:59.572073 27951 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260929 03:17:59.573118 27951 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P bce5b828f0a947c288a8cb13eb6c6819: tablet copy: Starting download of 0 data blocks...
I20260929 03:17:59.573282 27951 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P bce5b828f0a947c288a8cb13eb6c6819: tablet copy: Starting download of 1 WAL segments...
I20260929 03:17:59.573958 27951 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P bce5b828f0a947c288a8cb13eb6c6819: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260929 03:17:59.574652 27318 tablet_copy_service.cc:342] P d4b1269455e04063b0eb4e2aa46d0684: Request end of tablet copy session bce5b828f0a947c288a8cb13eb6c6819-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.20.178.124@KRBTEST.COM'} at 127.0.0.1:42482
I20260929 03:17:59.574705 27318 tablet_copy_service.cc:433] P d4b1269455e04063b0eb4e2aa46d0684: ending tablet copy session bce5b828f0a947c288a8cb13eb6c6819-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer bce5b828f0a947c288a8cb13eb6c6819
I20260929 03:17:59.576905 27951 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:17:59.578061 28055 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:17:59.578228 28056 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:17:59.578285 27951 server_base.cc:1034] running on GCE node
W20260929 03:17:59.578647 28058 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:17:59.578826 27951 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:17:59.578958 27951 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:17:59.580077 27951 hybrid_clock.cc:648] HybridClock initialized: now 1790651879580071 us; error 25 us; skew 500 ppm
Sep 29 03:17:59 dist-test-slave-c9qh krb5kdc[27253](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651879, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.178.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:17:59.581475 27951 init.cc:377] Logged in from keytab as oryx/127.20.178.124@KRBTEST.COM (short username oryx)
I20260929 03:17:59.581743 27951 webserver.cc:460] Webserver started at http://127.20.178.124:41489/ using document root <none> and password file <none>
I20260929 03:17:59.581857 27951 fs_manager.cc:359] Metadata directory not provided
I20260929 03:17:59.581892 27951 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:17:59.582495 27951 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.001s
I20260929 03:17:59.582993 28065 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:17:59.583099 27951 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260929 03:17:59.583137 27951 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/minicluster-data/master-2/data,/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/minicluster-data/master-2/wal
uuid: "bce5b828f0a947c288a8cb13eb6c6819"
format_stamp: "Formatted at 2026-09-29 03:17:59 on dist-test-slave-c9qh"
server_key: "869608a61231f923333812312e848cc0"
server_key_iv: "058c06acb041fe2ff3f4d5a9e07ba83e"
server_key_version: "encryptionkey@0"
I20260929 03:17:59.583190 27951 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:17:59.606469 27951 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:17:59.608210 27951 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:17:59.611378 27951 rpc_server.cc:307] RPC server started. Bound to: 127.20.178.124:41439
I20260929 03:17:59.611759 27951 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/minicluster-data/master-2/data/info.pb
I20260929 03:17:59.612752 28117 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.178.124:41439 every 8 connection(s)
I20260929 03:17:59.613408 28118 sys_catalog.cc:263] Verifying existing consensus state
I20260929 03:17:59.613972 28118 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bce5b828f0a947c288a8cb13eb6c6819: Bootstrap starting.
I20260929 03:17:59.616259 28118 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bce5b828f0a947c288a8cb13eb6c6819: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 03:17:59.616501 28118 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bce5b828f0a947c288a8cb13eb6c6819: Bootstrap complete.
I20260929 03:17:59.616730 28118 raft_consensus.cc:348] T 00000000000000000000000000000000 P bce5b828f0a947c288a8cb13eb6c6819 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d4b1269455e04063b0eb4e2aa46d0684" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 35553 } } peers { permanent_uuid: "82e1ecef80f449029732e3ade3c4a092" member_type: VOTER last_known_addr { host: "127.20.178.125" port: 35183 } attrs { promote: false } } peers { permanent_uuid: "bce5b828f0a947c288a8cb13eb6c6819" member_type: NON_VOTER last_known_addr { host: "127.20.178.124" port: 41439 } attrs { promote: true } }
I20260929 03:17:59.616819 28118 raft_consensus.cc:729] T 00000000000000000000000000000000 P bce5b828f0a947c288a8cb13eb6c6819 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: bce5b828f0a947c288a8cb13eb6c6819, State: Initialized, Role: LEARNER
I20260929 03:17:59.616864 28118 consensus_queue.cc:260] T 00000000000000000000000000000000 P bce5b828f0a947c288a8cb13eb6c6819 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d4b1269455e04063b0eb4e2aa46d0684" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 35553 } } peers { permanent_uuid: "82e1ecef80f449029732e3ade3c4a092" member_type: VOTER last_known_addr { host: "127.20.178.125" port: 35183 } attrs { promote: false } } peers { permanent_uuid: "bce5b828f0a947c288a8cb13eb6c6819" member_type: NON_VOTER last_known_addr { host: "127.20.178.124" port: 41439 } attrs { promote: true } }
I20260929 03:17:59.617045 28118 sys_catalog.cc:564] T 00000000000000000000000000000000 P bce5b828f0a947c288a8cb13eb6c6819 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:17:59.617041 28121 sys_catalog.cc:455] T 00000000000000000000000000000000 P bce5b828f0a947c288a8cb13eb6c6819 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d4b1269455e04063b0eb4e2aa46d0684" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 35553 } } peers { permanent_uuid: "82e1ecef80f449029732e3ade3c4a092" member_type: VOTER last_known_addr { host: "127.20.178.125" port: 35183 } attrs { promote: false } } peers { permanent_uuid: "bce5b828f0a947c288a8cb13eb6c6819" member_type: NON_VOTER last_known_addr { host: "127.20.178.124" port: 41439 } attrs { promote: true } } }
I20260929 03:17:59.617295 28121 sys_catalog.cc:458] T 00000000000000000000000000000000 P bce5b828f0a947c288a8cb13eb6c6819 [sys.catalog]: This master's current role is: LEARNER
I20260929 03:17:59.618520 28132 catalog_manager.cc:1259] Loaded cluster ID: d4b12290c9754a30a1ed0ccfd1dc774e
I20260929 03:17:59.618602 28132 catalog_manager.cc:1552] T 00000000000000000000000000000000 P bce5b828f0a947c288a8cb13eb6c6819: loading cluster ID for follower catalog manager: success
I20260929 03:17:59.619789 28132 catalog_manager.cc:1574] T 00000000000000000000000000000000 P bce5b828f0a947c288a8cb13eb6c6819: acquiring CA information for follower catalog manager: success
I20260929 03:17:59.620059 28132 catalog_manager.cc:1602] T 00000000000000000000000000000000 P bce5b828f0a947c288a8cb13eb6c6819: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260929 03:17:59.664826 28044 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:17:59.665078 28044 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:17:59.665114 28044 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:17:59.666738 28044 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:17:59.666793 28044 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:17:59.666829 28044 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.178.65
I20260929 03:17:59.668876 28044 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.178.84:35629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/krb5kdc/oryx_127.20.178.65.keytab
--principal=oryx/127.20.178.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.178.65:45139
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.178.65
--webserver_port=36929
--tserver_master_addrs=127.20.178.126:35553,127.20.178.125:35183,127.20.178.124:41439
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28044
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.178.65
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:17:59.669101 28044 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:17:59.669332 28044 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:17:59.674345 28137 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:17:59.674398 28140 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:17:59.674594 28138 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:17:59.674782 28044 server_base.cc:1034] running on GCE node
I20260929 03:17:59.674986 28044 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:17:59.675226 28044 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:17:59.676386 28044 hybrid_clock.cc:648] HybridClock initialized: now 1790651879676371 us; error 30 us; skew 500 ppm
Sep 29 03:17:59 dist-test-slave-c9qh krb5kdc[27253](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651879, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.178.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:17:59.678982 28044 init.cc:377] Logged in from keytab as oryx/127.20.178.65@KRBTEST.COM (short username oryx)
I20260929 03:17:59.679435 28044 webserver.cc:460] Webserver started at http://127.20.178.65:36929/ using document root <none> and password file <none>
I20260929 03:17:59.679663 28044 fs_manager.cc:359] Metadata directory not provided
I20260929 03:17:59.679716 28044 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:17:59.681113 28044 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260929 03:17:59.681695 28147 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:17:59.681854 28044 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 03:17:59.681948 28044 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/minicluster-data/ts-0/data,/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/minicluster-data/ts-0/wal
uuid: "e76cae1a85be4646b41cc2ed226fe5c7"
format_stamp: "Formatted at 2026-09-29 03:17:58 on dist-test-slave-c9qh"
server_key: "15d4b51628ea93bffd5ebff7517c9500"
server_key_iv: "3c2e60dfdc4fc18b3813c00ab4e59740"
server_key_version: "encryptionkey@0"
I20260929 03:17:59.682277 28044 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:17:59.693058 28044 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:17:59.695356 28044 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:17:59.695510 28044 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:17:59.695739 28044 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:17:59.696106 28044 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:17:59.696164 28044 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:17:59.696205 28044 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:17:59.696231 28044 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:17:59.702355 28044 rpc_server.cc:307] RPC server started. Bound to: 127.20.178.65:45139
I20260929 03:17:59.702410 28260 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.178.65:45139 every 8 connection(s)
I20260929 03:17:59.702724 28044 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/minicluster-data/ts-0/data/info.pb
Sep 29 03:17:59 dist-test-slave-c9qh krb5kdc[27253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651879, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.178.65@KRBTEST.COM for oryx/127.20.178.125@KRBTEST.COM
Sep 29 03:17:59 dist-test-slave-c9qh krb5kdc[27253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651879, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.178.65@KRBTEST.COM for oryx/127.20.178.124@KRBTEST.COM
Sep 29 03:17:59 dist-test-slave-c9qh krb5kdc[27253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651879, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.178.65@KRBTEST.COM for oryx/127.20.178.126@KRBTEST.COM
I20260929 03:17:59.713433 21193 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuoaHbX/build/release/bin/kudu as pid 28044
I20260929 03:17:59.713608 21193 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuoaHbX/build/release/bin/kudu with pid 27494
I20260929 03:17:59.713987 27948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:59.703786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.65:49059 (local address 127.20.178.125:35183)
0929 03:17:59.703844 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:59.703846 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:59.703854 (+ 8us) server_negotiation.cc:407] Connection header received
0929 03:17:59.704351 (+ 497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:59.704353 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:59.704365 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:59.704435 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:17:59.705274 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:59.706032 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:59.707330 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:59.707656 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:59.707721 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:59.710342 (+ 2621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:17:59.710343 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:17:59.710344 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:17:59.710354 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:17:59.712126 (+ 1772us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:59.712697 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:59.712699 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:59.712699 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:59.712764 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:59.713093 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:59.713094 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:59.713094 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:59.713251 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:17:59.713316 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0929 03:17:59.713692 (+ 376us) server_negotiation.cc:299] Negotiation successful
0929 03:17:59.713772 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260929 03:17:59.714502 28266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:59.703493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.125:35183 (local address 127.20.178.65:49059)
0929 03:17:59.703726 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0929 03:17:59.703735 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 03:17:59.704234 (+ 499us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:17:59.704502 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:17:59.704506 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:17:59.704955 (+ 449us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:17:59.705161 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:59.705167 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:59.706124 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:59.706124 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:59.707257 (+ 1133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:59.707260 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:59.707738 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:59.707740 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:59.707842 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:59.708618 (+ 776us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:17:59.708669 (+ 51us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:17:59.710215 (+ 1546us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:17:59.712211 (+ 1996us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:17:59.712217 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:17:59.712219 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:17:59.712587 (+ 368us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:17:59.712848 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:17:59.712849 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:17:59.712849 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:17:59.713005 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:17:59.713343 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:17:59.713349 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:17:59.713508 (+ 159us) client_negotiation.cc:769] Sending connection context
0929 03:17:59.714093 (+ 585us) client_negotiation.cc:240] Negotiation successful
0929 03:17:59.714263 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209,"mutex_wait_us":124,"thread_start_us":148,"threads_started":1}
I20260929 03:17:59.714978 28263 heartbeater.cc:347] Connected to a master server at 127.20.178.125:35183
I20260929 03:17:59.715082 28263 heartbeater.cc:464] Registering TS with master...
I20260929 03:17:59.715214 27356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:59.703484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.65:56051 (local address 127.20.178.126:35553)
0929 03:17:59.703533 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:59.703536 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:59.703800 (+ 264us) server_negotiation.cc:407] Connection header received
0929 03:17:59.704354 (+ 554us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:59.704356 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:59.704370 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:59.704400 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:17:59.705274 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:59.706025 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:59.707395 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:59.707602 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:59.707719 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:59.711282 (+ 3563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:17:59.711283 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:17:59.711283 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:17:59.711291 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:17:59.713155 (+ 1864us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:59.713851 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:59.713852 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:59.713852 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:59.713909 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:59.714377 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:59.714378 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:59.714378 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:59.714510 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:17:59.714556 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0929 03:17:59.715006 (+ 450us) server_negotiation.cc:299] Negotiation successful
0929 03:17:59.715104 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260929 03:17:59.715318 28263 heartbeater.cc:511] Master 127.20.178.125:35183 requested a full tablet report, sending...
I20260929 03:17:59.715544 28265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:59.703493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:35553 (local address 127.20.178.65:56051)
0929 03:17:59.703649 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0929 03:17:59.703713 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0929 03:17:59.704238 (+ 525us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:17:59.704502 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:17:59.704506 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:17:59.704960 (+ 454us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:17:59.705160 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:59.705167 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:59.706124 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:59.706124 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:59.707301 (+ 1177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:59.707303 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:59.707773 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:59.707774 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:59.707834 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:59.708525 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:17:59.708529 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:17:59.711176 (+ 2647us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:17:59.713249 (+ 2073us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:17:59.713250 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:17:59.713250 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:17:59.713768 (+ 518us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:17:59.713981 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:17:59.713981 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:17:59.713981 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:17:59.714292 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:17:59.714842 (+ 550us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:17:59.714843 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:17:59.714888 (+ 45us) client_negotiation.cc:769] Sending connection context
0929 03:17:59.715339 (+ 451us) client_negotiation.cc:240] Negotiation successful
0929 03:17:59.715438 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"mutex_wait_us":84,"thread_start_us":95,"threads_started":1}
I20260929 03:17:59.715639 28268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:59.703925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.65:41733 (local address 127.20.178.124:41439)
0929 03:17:59.704055 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:59.704058 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:59.704104 (+ 46us) server_negotiation.cc:407] Connection header received
0929 03:17:59.704500 (+ 396us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:59.704501 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:59.704513 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:59.704611 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:17:59.705263 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:59.705995 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:59.707248 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:59.707589 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:59.707662 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:59.711502 (+ 3840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:17:59.711503 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:17:59.711503 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:17:59.711510 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:17:59.713165 (+ 1655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:59.713644 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:59.713646 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:59.713646 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:59.713711 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:59.713924 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:59.713925 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:59.713926 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:59.714125 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:17:59.714188 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0929 03:17:59.714734 (+ 546us) server_negotiation.cc:299] Negotiation successful
0929 03:17:59.714842 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":49,"threads_started":1}
I20260929 03:17:59.715870 27897 ts_manager.cc:194] Registered new tserver with Master: e76cae1a85be4646b41cc2ed226fe5c7 (127.20.178.65:45139)
I20260929 03:17:59.715885 28267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:59.703827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.124:41439 (local address 127.20.178.65:41733)
0929 03:17:59.704043 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0929 03:17:59.704055 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 03:17:59.704412 (+ 357us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:17:59.704616 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:17:59.704617 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:17:59.704931 (+ 314us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:17:59.705160 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:59.705165 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:59.706115 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:59.706116 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:59.707137 (+ 1021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:59.707139 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:59.707677 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:59.707678 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:59.707816 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:59.708509 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:17:59.708519 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:17:59.711411 (+ 2892us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:17:59.713249 (+ 1838us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:17:59.713250 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:17:59.713251 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:17:59.713567 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:17:59.713791 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:17:59.713792 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:17:59.713793 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:17:59.713857 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:17:59.714217 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:17:59.714218 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:17:59.714265 (+ 47us) client_negotiation.cc:769] Sending connection context
0929 03:17:59.714336 (+ 71us) client_negotiation.cc:240] Negotiation successful
0929 03:17:59.714459 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"mutex_wait_us":112,"thread_start_us":65,"threads_started":1}
I20260929 03:17:59.719738 28262 heartbeater.cc:347] Connected to a master server at 127.20.178.126:35553
I20260929 03:17:59.719794 28262 heartbeater.cc:464] Registering TS with master...
I20260929 03:17:59.719866 28262 heartbeater.cc:511] Master 127.20.178.126:35553 requested a full tablet report, sending...
I20260929 03:17:59.720034 28261 heartbeater.cc:347] Connected to a master server at 127.20.178.124:41439
I20260929 03:17:59.720100 28261 heartbeater.cc:464] Registering TS with master...
I20260929 03:17:59.720147 27298 ts_manager.cc:194] Re-registered known tserver with Master: e76cae1a85be4646b41cc2ed226fe5c7 (127.20.178.65:45139)
I20260929 03:17:59.720185 28261 heartbeater.cc:511] Master 127.20.178.124:41439 requested a full tablet report, sending...
I20260929 03:17:59.720572 27298 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.20.178.65@KRBTEST.COM'} at 127.20.178.65:56051
I20260929 03:17:59.720561 28082 ts_manager.cc:194] Registered new tserver with Master: e76cae1a85be4646b41cc2ed226fe5c7 (127.20.178.65:45139)
I20260929 03:17:59.720729 21193 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuoaHbX/build/release/bin/kudu
/tmp/dist-test-taskuoaHbX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.178.66:42401
--local_ip_for_outbound_sockets=127.20.178.66
--tserver_master_addrs=127.20.178.126:35553
--webserver_port=41939
--webserver_interface=127.20.178.66
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/krb5kdc/oryx_127.20.178.66.keytab
--principal=oryx/127.20.178.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.178.84:35629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.20.178.126:35553,127.20.178.125:35183
--tserver_master_addrs=127.20.178.126:35553,127.20.178.125:35183,127.20.178.124:41439 with env {KRB5CCNAME=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:17:59.822295 28269 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:17:59.822512 28269 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:17:59.822537 28269 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:17:59.823824 28269 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:17:59.823895 28269 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:17:59.823954 28269 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.178.66
I20260929 03:17:59.825786 28269 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.178.84:35629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/krb5kdc/oryx_127.20.178.66.keytab
--principal=oryx/127.20.178.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.178.66:42401
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.178.66
--webserver_port=41939
--tserver_master_addrs=127.20.178.126:35553,127.20.178.125:35183,127.20.178.124:41439
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28269
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.178.66
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:17:59.825984 28269 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:17:59.826169 28269 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:17:59.831054 28274 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:17:59.831138 28277 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:17:59.831075 28275 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:17:59.831353 28269 server_base.cc:1034] running on GCE node
I20260929 03:17:59.831588 28269 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:17:59.831839 28269 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:17:59.832988 28269 hybrid_clock.cc:648] HybridClock initialized: now 1790651879832973 us; error 31 us; skew 500 ppm
Sep 29 03:17:59 dist-test-slave-c9qh krb5kdc[27253](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651879, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.178.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:17:59.835398 28269 init.cc:377] Logged in from keytab as oryx/127.20.178.66@KRBTEST.COM (short username oryx)
I20260929 03:17:59.835843 28269 webserver.cc:460] Webserver started at http://127.20.178.66:41939/ using document root <none> and password file <none>
I20260929 03:17:59.836076 28269 fs_manager.cc:359] Metadata directory not provided
I20260929 03:17:59.836159 28269 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:17:59.837538 28269 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 03:17:59.838137 28284 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:17:59.838331 28269 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 03:17:59.838433 28269 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/minicluster-data/ts-1/data,/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/minicluster-data/ts-1/wal
uuid: "5064f9a44a034c0dadb4054ac8824780"
format_stamp: "Formatted at 2026-09-29 03:17:58 on dist-test-slave-c9qh"
server_key: "eed6d3c0ad3474a10b2e7d435b78af62"
server_key_iv: "f24a1a875cb2a1c09df321bdff4c6df8"
server_key_version: "encryptionkey@0"
I20260929 03:17:59.838785 28269 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:17:59.848109 28269 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:17:59.849872 28269 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:17:59.850029 28269 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:17:59.850231 28269 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:17:59.850533 28269 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:17:59.850589 28269 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:17:59.850629 28269 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:17:59.850665 28269 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:17:59.856444 28269 rpc_server.cc:307] RPC server started. Bound to: 127.20.178.66:42401
I20260929 03:17:59.856510 28397 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.178.66:42401 every 8 connection(s)
I20260929 03:17:59.856885 28269 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/minicluster-data/ts-1/data/info.pb
Sep 29 03:17:59 dist-test-slave-c9qh krb5kdc[27253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651879, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.178.66@KRBTEST.COM for oryx/127.20.178.126@KRBTEST.COM
Sep 29 03:17:59 dist-test-slave-c9qh krb5kdc[27253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651879, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.178.66@KRBTEST.COM for oryx/127.20.178.125@KRBTEST.COM
Sep 29 03:17:59 dist-test-slave-c9qh krb5kdc[27253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651879, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.178.66@KRBTEST.COM for oryx/127.20.178.124@KRBTEST.COM
I20260929 03:17:59.865885 21193 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuoaHbX/build/release/bin/kudu as pid 28269
I20260929 03:17:59.866052 21193 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuoaHbX/build/release/bin/kudu with pid 27628
I20260929 03:17:59.867448 27356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:59.857581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.66:54841 (local address 127.20.178.126:35553)
0929 03:17:59.857831 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:59.857833 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:59.857899 (+ 66us) server_negotiation.cc:407] Connection header received
0929 03:17:59.859112 (+ 1213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:59.859114 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:59.859127 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:59.859185 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:17:59.859846 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:59.860352 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:59.861427 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:59.861636 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:59.861719 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:59.863984 (+ 2265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:17:59.863986 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:17:59.863986 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:17:59.863994 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:17:59.865843 (+ 1849us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:59.866324 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:59.866325 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:59.866326 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:59.866385 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:59.866667 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:59.866668 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:59.866668 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:59.866807 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:17:59.866855 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0929 03:17:59.867245 (+ 390us) server_negotiation.cc:299] Negotiation successful
0929 03:17:59.867331 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":225}
I20260929 03:17:59.868083 28402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:59.857582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:35553 (local address 127.20.178.66:54841)
0929 03:17:59.857770 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0929 03:17:59.857815 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 03:17:59.858993 (+ 1178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:17:59.859270 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:17:59.859273 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:17:59.859557 (+ 284us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:17:59.859749 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:59.859754 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:59.860460 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:59.860461 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:59.861327 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:59.861329 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:59.861730 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:59.861731 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:59.861868 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:59.862403 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:17:59.862413 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:17:59.863893 (+ 1480us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:17:59.865935 (+ 2042us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:17:59.865941 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:17:59.865943 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:17:59.866231 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:17:59.866458 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:17:59.866459 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:17:59.866459 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:17:59.866591 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:17:59.866908 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:17:59.866914 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:17:59.867074 (+ 160us) client_negotiation.cc:769] Sending connection context
0929 03:17:59.867592 (+ 518us) client_negotiation.cc:240] Negotiation successful
0929 03:17:59.867856 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":117,"threads_started":1}
I20260929 03:17:59.868512 28399 heartbeater.cc:347] Connected to a master server at 127.20.178.126:35553
I20260929 03:17:59.868604 28399 heartbeater.cc:464] Registering TS with master...
I20260929 03:17:59.868820 27948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:59.857866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.66:59113 (local address 127.20.178.125:35183)
0929 03:17:59.857910 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:59.857912 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:59.858053 (+ 141us) server_negotiation.cc:407] Connection header received
0929 03:17:59.859134 (+ 1081us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:59.859136 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:59.859149 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:59.859183 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:17:59.859855 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:59.860608 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:59.861759 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:59.862045 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:59.862086 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:59.865101 (+ 3015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:17:59.865103 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:17:59.865103 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:17:59.865111 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:17:59.867428 (+ 2317us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:59.867938 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:59.867939 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:59.867939 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:59.867998 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:59.868230 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:59.868231 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:59.868232 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:59.868362 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:17:59.868438 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0929 03:17:59.868646 (+ 208us) server_negotiation.cc:299] Negotiation successful
0929 03:17:59.868733 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260929 03:17:59.868822 28399 heartbeater.cc:511] Master 127.20.178.126:35553 requested a full tablet report, sending...
I20260929 03:17:59.869002 28403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:59.857617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.125:35183 (local address 127.20.178.66:59113)
0929 03:17:59.857994 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0929 03:17:59.858002 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0929 03:17:59.859023 (+ 1021us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:17:59.859285 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:17:59.859286 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:17:59.859612 (+ 326us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:17:59.859749 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:59.859757 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:59.860732 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:59.860733 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:59.861659 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:59.861661 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:59.862163 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:59.862164 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:59.862225 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:59.862774 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:17:59.862778 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:17:59.865008 (+ 2230us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:17:59.867569 (+ 2561us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:17:59.867570 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:17:59.867571 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:17:59.867851 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:17:59.868065 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:17:59.868066 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:17:59.868066 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:17:59.868122 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:17:59.868470 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:17:59.868470 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:17:59.868512 (+ 42us) client_negotiation.cc:769] Sending connection context
0929 03:17:59.868580 (+ 68us) client_negotiation.cc:240] Negotiation successful
0929 03:17:59.868696 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":356,"mutex_wait_us":37,"thread_start_us":87,"threads_started":1}
I20260929 03:17:59.869256 27298 ts_manager.cc:194] Re-registered known tserver with Master: 5064f9a44a034c0dadb4054ac8824780 (127.20.178.66:42401)
I20260929 03:17:59.869568 27298 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.20.178.66@KRBTEST.COM'} at 127.20.178.66:54841
I20260929 03:17:59.873564 28400 heartbeater.cc:347] Connected to a master server at 127.20.178.125:35183
I20260929 03:17:59.873661 28400 heartbeater.cc:464] Registering TS with master...
I20260929 03:17:59.873755 28400 heartbeater.cc:511] Master 127.20.178.125:35183 requested a full tablet report, sending...
I20260929 03:17:59.874116 27897 ts_manager.cc:194] Registered new tserver with Master: 5064f9a44a034c0dadb4054ac8824780 (127.20.178.66:42401)
I20260929 03:17:59.874351 21193 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuoaHbX/build/release/bin/kudu
/tmp/dist-test-taskuoaHbX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.178.67:32833
--local_ip_for_outbound_sockets=127.20.178.67
--tserver_master_addrs=127.20.178.126:35553
--webserver_port=40205
--webserver_interface=127.20.178.67
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/krb5kdc/oryx_127.20.178.67.keytab
--principal=oryx/127.20.178.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.178.84:35629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.20.178.126:35553,127.20.178.125:35183
--tserver_master_addrs=127.20.178.126:35553,127.20.178.125:35183,127.20.178.124:41439 with env {KRB5CCNAME=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 03:17:59.875960 28268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:59.857868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.66:46859 (local address 127.20.178.124:41439)
0929 03:17:59.857910 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:59.857912 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:59.858077 (+ 165us) server_negotiation.cc:407] Connection header received
0929 03:17:59.859134 (+ 1057us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:59.859136 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:59.859152 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:59.859190 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:17:59.859857 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:59.860614 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:59.861759 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:59.862068 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:59.862103 (+ 35us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:59.865274 (+ 3171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:17:59.865276 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:17:59.865276 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:17:59.865284 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:17:59.874719 (+ 9435us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:59.875110 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:59.875111 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:59.875111 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:59.875186 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:59.875363 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:59.875364 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:59.875364 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:59.875513 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:17:59.875581 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0929 03:17:59.875806 (+ 225us) server_negotiation.cc:299] Negotiation successful
0929 03:17:59.875897 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260929 03:17:59.876048 28404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:59.857583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.124:41439 (local address 127.20.178.66:46859)
0929 03:17:59.857981 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0929 03:17:59.857992 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 03:17:59.859027 (+ 1035us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:17:59.859285 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:17:59.859286 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:17:59.859614 (+ 328us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:17:59.859749 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:59.859757 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:59.860732 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:59.860733 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:59.861659 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:59.861661 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:59.862168 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:59.862169 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:59.862231 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:59.862703 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:17:59.862707 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:17:59.865146 (+ 2439us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:17:59.874802 (+ 9656us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:17:59.874803 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:17:59.874803 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:17:59.875038 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:17:59.875254 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:17:59.875254 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:17:59.875254 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:17:59.875304 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:17:59.875573 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:17:59.875574 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:17:59.875637 (+ 63us) client_negotiation.cc:769] Sending connection context
0929 03:17:59.875765 (+ 128us) client_negotiation.cc:240] Negotiation successful
0929 03:17:59.875901 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":372,"mutex_wait_us":97,"thread_start_us":104,"threads_started":1}
I20260929 03:17:59.876372 28398 heartbeater.cc:347] Connected to a master server at 127.20.178.124:41439
I20260929 03:17:59.876426 28398 heartbeater.cc:464] Registering TS with master...
I20260929 03:17:59.876479 28398 heartbeater.cc:511] Master 127.20.178.124:41439 requested a full tablet report, sending...
I20260929 03:17:59.876807 28082 ts_manager.cc:194] Registered new tserver with Master: 5064f9a44a034c0dadb4054ac8824780 (127.20.178.66:42401)
W20260929 03:17:59.955595 28405 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:17:59.955823 28405 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:17:59.955863 28405 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260929 03:17:59.956990 27856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:59.953989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.124:41439 (local address 127.0.0.1:44062)
0929 03:17:59.954061 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0929 03:17:59.954094 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0929 03:17:59.954186 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:17:59.954379 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:17:59.954380 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:17:59.954382 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:17:59.954435 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:59.954437 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:59.955141 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:59.955143 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:59.955902 (+ 759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:59.955905 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:59.956777 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:59.956778 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:59.956835 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:59.956838 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 03:17:59.956893 (+ 55us) client_negotiation.cc:240] Negotiation successful
0929 03:17:59.956933 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260929 03:17:59.957027 28268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:59.954103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44062 (local address 127.20.178.124:41439)
0929 03:17:59.954175 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:59.954177 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:59.954189 (+ 12us) server_negotiation.cc:407] Connection header received
0929 03:17:59.954261 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:59.954262 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:59.954287 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:59.954360 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:17:59.954516 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:59.955045 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:59.956008 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:59.956627 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:59.956731 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:59.956804 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 03:17:59.956913 (+ 109us) server_negotiation.cc:299] Negotiation successful
0929 03:17:59.956959 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 03:17:59.957304 28092 raft_consensus.cc:1203] T 00000000000000000000000000000000 P bce5b828f0a947c288a8cb13eb6c6819 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260929 03:17:59.957528 28121 sys_catalog.cc:455] T 00000000000000000000000000000000 P bce5b828f0a947c288a8cb13eb6c6819 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d4b1269455e04063b0eb4e2aa46d0684. Latest consensus state: current_term: 1 leader_uuid: "d4b1269455e04063b0eb4e2aa46d0684" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d4b1269455e04063b0eb4e2aa46d0684" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 35553 } } peers { permanent_uuid: "82e1ecef80f449029732e3ade3c4a092" member_type: VOTER last_known_addr { host: "127.20.178.125" port: 35183 } attrs { promote: false } } peers { permanent_uuid: "bce5b828f0a947c288a8cb13eb6c6819" member_type: NON_VOTER last_known_addr { host: "127.20.178.124" port: 41439 } attrs { promote: true } } }
W20260929 03:17:59.957584 28405 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:17:59.957638 28405 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
I20260929 03:17:59.957635 28121 sys_catalog.cc:458] T 00000000000000000000000000000000 P bce5b828f0a947c288a8cb13eb6c6819 [sys.catalog]: This master's current role is: LEARNER
W20260929 03:17:59.957692 28405 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.178.67
I20260929 03:17:59.959146 28405 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.178.84:35629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/krb5kdc/oryx_127.20.178.67.keytab
--principal=oryx/127.20.178.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.178.67:32833
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.178.67
--webserver_port=40205
--tserver_master_addrs=127.20.178.126:35553,127.20.178.125:35183,127.20.178.124:41439
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28405
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.178.67
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:17:59.959370 28405 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:17:59.959586 28405 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:17:59.964598 28413 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:17:59.964620 28411 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:17:59.964845 28405 server_base.cc:1034] running on GCE node
W20260929 03:17:59.964946 28410 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:17:59.965140 28405 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:17:59.965431 28405 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:17:59.966594 28405 hybrid_clock.cc:648] HybridClock initialized: now 1790651879966573 us; error 38 us; skew 500 ppm
Sep 29 03:17:59 dist-test-slave-c9qh krb5kdc[27253](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651879, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.178.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:17:59.969100 28405 init.cc:377] Logged in from keytab as oryx/127.20.178.67@KRBTEST.COM (short username oryx)
I20260929 03:17:59.969538 28405 webserver.cc:460] Webserver started at http://127.20.178.67:40205/ using document root <none> and password file <none>
I20260929 03:17:59.969766 28405 fs_manager.cc:359] Metadata directory not provided
I20260929 03:17:59.969846 28405 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:17:59.971069 28405 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 03:17:59.971581 28420 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:17:59.971750 28405 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260929 03:17:59.971841 28405 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/minicluster-data/ts-2/data,/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/minicluster-data/ts-2/wal
uuid: "b2e7ad7ddf214b3596c571f5cd4d3272"
format_stamp: "Formatted at 2026-09-29 03:17:59 on dist-test-slave-c9qh"
server_key: "a1b2520717078d0eb5d17adc8311d1a5"
server_key_iv: "4bab638e4583baef20b315406a7b38cf"
server_key_version: "encryptionkey@0"
I20260929 03:17:59.972162 28405 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:17:59.986313 28405 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:17:59.988009 28405 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:17:59.988157 28405 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:17:59.988363 28405 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:17:59.988695 28405 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:17:59.988757 28405 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:17:59.988811 28405 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:17:59.988838 28405 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:17:59.994624 28405 rpc_server.cc:307] RPC server started. Bound to: 127.20.178.67:32833
I20260929 03:17:59.994666 28533 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.178.67:32833 every 8 connection(s)
I20260929 03:17:59.994976 28405 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/minicluster-data/ts-2/data/info.pb
I20260929 03:17:59.999890 21193 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuoaHbX/build/release/bin/kudu as pid 28405
Sep 29 03:17:59 dist-test-slave-c9qh krb5kdc[27253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651879, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.178.67@KRBTEST.COM for oryx/127.20.178.124@KRBTEST.COM
Sep 29 03:17:59 dist-test-slave-c9qh krb5kdc[27253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651879, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.178.67@KRBTEST.COM for oryx/127.20.178.125@KRBTEST.COM
Sep 29 03:17:59 dist-test-slave-c9qh krb5kdc[27253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651879, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.178.67@KRBTEST.COM for oryx/127.20.178.126@KRBTEST.COM
I20260929 03:18:00.007615 28538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:59.995627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.124:41439 (local address 127.20.178.67:41125)
0929 03:17:59.996752 (+ 1125us) negotiation.cc:107] Waiting for socket to connect
0929 03:17:59.996764 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 03:17:59.997387 (+ 623us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:17:59.997617 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:17:59.997621 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:17:59.997980 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:17:59.998093 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:59.998095 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:59.999093 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:59.999094 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:59.999994 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:59.999997 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:00.000412 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:00.000414 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:00.000490 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:00.001060 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:18:00.001065 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:18:00.003988 (+ 2923us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:18:00.005913 (+ 1925us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:18:00.005919 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:18:00.005920 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:18:00.006205 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:18:00.006468 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:18:00.006469 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:18:00.006469 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:18:00.006604 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:18:00.006931 (+ 327us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:18:00.006937 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:18:00.007113 (+ 176us) client_negotiation.cc:769] Sending connection context
0929 03:18:00.007227 (+ 114us) client_negotiation.cc:240] Negotiation successful
0929 03:18:00.007395 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1099,"mutex_wait_us":52,"thread_start_us":89,"threads_started":1}
I20260929 03:18:00.008021 28268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:59.995851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.67:41125 (local address 127.20.178.124:41439)
0929 03:17:59.995901 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:59.995903 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:59.996844 (+ 941us) server_negotiation.cc:407] Connection header received
0929 03:17:59.997497 (+ 653us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:59.997498 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:59.997515 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:59.997546 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:17:59.998208 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:59.998987 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:00.000068 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:00.000336 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:00.000400 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:00.004065 (+ 3665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:18:00.004066 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:18:00.004067 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:18:00.004073 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:18:00.005833 (+ 1760us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:18:00.006330 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:18:00.006332 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:18:00.006332 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:18:00.006393 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:18:00.006693 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:18:00.006694 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:18:00.006694 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:18:00.006833 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:18:00.006903 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0929 03:18:00.007833 (+ 930us) server_negotiation.cc:299] Negotiation successful
0929 03:18:00.007921 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260929 03:18:00.008374 28534 heartbeater.cc:347] Connected to a master server at 127.20.178.124:41439
I20260929 03:18:00.008461 28534 heartbeater.cc:464] Registering TS with master...
I20260929 03:18:00.008744 28534 heartbeater.cc:511] Master 127.20.178.124:41439 requested a full tablet report, sending...
I20260929 03:18:00.009230 28082 ts_manager.cc:194] Registered new tserver with Master: b2e7ad7ddf214b3596c571f5cd4d3272 (127.20.178.67:32833)
I20260929 03:18:00.009583 28539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:59.995626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:35553 (local address 127.20.178.67:45701)
0929 03:17:59.996102 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0929 03:17:59.996145 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 03:17:59.997378 (+ 1233us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:17:59.997617 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:17:59.997621 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:17:59.997983 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:17:59.998093 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:59.998095 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:59.999093 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:59.999094 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:00.001093 (+ 1999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:00.001096 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:00.001906 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:00.001907 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:00.001978 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:00.002376 (+ 398us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:18:00.002380 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:18:00.004301 (+ 1921us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:18:00.008070 (+ 3769us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:18:00.008071 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:18:00.008071 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:18:00.008424 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:18:00.008678 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:18:00.008679 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:18:00.008679 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:18:00.008738 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:18:00.009033 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:18:00.009034 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:18:00.009076 (+ 42us) client_negotiation.cc:769] Sending connection context
0929 03:18:00.009178 (+ 102us) client_negotiation.cc:240] Negotiation successful
0929 03:18:00.009284 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":437,"mutex_wait_us":37,"thread_start_us":107,"threads_started":1}
I20260929 03:18:00.009900 27356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:59.995798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.67:45701 (local address 127.20.178.126:35553)
0929 03:17:59.995933 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:59.995935 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:59.996228 (+ 293us) server_negotiation.cc:407] Connection header received
0929 03:17:59.997498 (+ 1270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:59.997499 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:59.997512 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:59.997544 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:17:59.998201 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:59.998968 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:00.001413 (+ 2445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:00.001679 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:00.001710 (+ 31us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:00.004780 (+ 3070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:18:00.004782 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:18:00.004782 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:18:00.004789 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:18:00.008003 (+ 3214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:18:00.008513 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:18:00.008514 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:18:00.008515 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:18:00.008577 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:18:00.008812 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:18:00.008813 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:18:00.008814 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:18:00.008947 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:18:00.009007 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0929 03:18:00.009718 (+ 711us) server_negotiation.cc:299] Negotiation successful
0929 03:18:00.009800 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109}
I20260929 03:18:00.010090 27948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:59.995979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.67:50261 (local address 127.20.178.125:35183)
0929 03:17:59.996049 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:59.996051 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:59.996828 (+ 777us) server_negotiation.cc:407] Connection header received
0929 03:17:59.997472 (+ 644us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:59.997473 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:59.997484 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:59.997539 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:17:59.998191 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:59.998658 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:59.999752 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:00.000044 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:00.000106 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:00.003593 (+ 3487us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:18:00.003594 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:18:00.003595 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:18:00.003602 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:18:00.007216 (+ 3614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:18:00.009213 (+ 1997us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:18:00.009215 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:18:00.009215 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:18:00.009276 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:18:00.009490 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:18:00.009491 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:18:00.009492 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:18:00.009644 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:18:00.009709 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0929 03:18:00.009891 (+ 182us) server_negotiation.cc:299] Negotiation successful
0929 03:18:00.009987 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260929 03:18:00.010352 28535 heartbeater.cc:347] Connected to a master server at 127.20.178.126:35553
I20260929 03:18:00.010406 28535 heartbeater.cc:464] Registering TS with master...
I20260929 03:18:00.010484 28535 heartbeater.cc:511] Master 127.20.178.126:35553 requested a full tablet report, sending...
I20260929 03:18:00.010495 28547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:00.001068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42490 (local address 127.20.178.126:35553)
0929 03:18:00.001384 (+ 316us) server_negotiation.cc:206] Beginning negotiation
0929 03:18:00.001386 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:18:00.001393 (+ 7us) server_negotiation.cc:407] Connection header received
0929 03:18:00.001412 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:18:00.001413 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:18:00.001424 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:18:00.001485 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:18:00.002542 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:00.003038 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:00.003882 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:00.004154 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:00.004474 (+ 320us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:00.005874 (+ 1400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:18:00.005875 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:18:00.005875 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:18:00.005882 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:18:00.008444 (+ 2562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:18:00.009175 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:18:00.009176 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:18:00.009176 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:18:00.009233 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:18:00.009445 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:18:00.009446 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:18:00.009446 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:18:00.009579 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:18:00.010272 (+ 693us) server_negotiation.cc:1035] Waiting for connection context
0929 03:18:00.010312 (+ 40us) server_negotiation.cc:299] Negotiation successful
0929 03:18:00.010387 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":83,"threads_started":1}
I20260929 03:18:00.010744 27297 ts_manager.cc:194] Re-registered known tserver with Master: b2e7ad7ddf214b3596c571f5cd4d3272 (127.20.178.67:32833)
I20260929 03:18:00.011193 27297 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.20.178.67@KRBTEST.COM'} at 127.20.178.67:45701
I20260929 03:18:00.011382 28540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:59.995627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.125:35183 (local address 127.20.178.67:50261)
0929 03:17:59.996757 (+ 1130us) negotiation.cc:107] Waiting for socket to connect
0929 03:17:59.996766 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 03:17:59.997380 (+ 614us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:17:59.997617 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:17:59.997620 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:17:59.997919 (+ 299us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:17:59.998091 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:59.998095 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:59.998806 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:59.998806 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:59.999656 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:59.999658 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:00.000218 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:00.000219 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:00.000351 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:00.001052 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:18:00.001062 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:18:00.003497 (+ 2435us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:18:00.008232 (+ 4735us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:18:00.008233 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:18:00.008233 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:18:00.009129 (+ 896us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:18:00.009352 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:18:00.009353 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:18:00.009353 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:18:00.009416 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:18:00.009746 (+ 330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:18:00.009747 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:18:00.009784 (+ 37us) client_negotiation.cc:769] Sending connection context
0929 03:18:00.010987 (+ 1203us) client_negotiation.cc:240] Negotiation successful
0929 03:18:00.011100 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1108,"mutex_wait_us":57,"thread_start_us":69,"threads_started":1}
I20260929 03:18:00.011718 28536 heartbeater.cc:347] Connected to a master server at 127.20.178.125:35183
I20260929 03:18:00.011768 28536 heartbeater.cc:464] Registering TS with master...
I20260929 03:18:00.011842 28536 heartbeater.cc:511] Master 127.20.178.125:35183 requested a full tablet report, sending...
I20260929 03:18:00.012147 27897 ts_manager.cc:194] Registered new tserver with Master: b2e7ad7ddf214b3596c571f5cd4d3272 (127.20.178.67:32833)
I20260929 03:18:00.014237 28545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:00.000925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44074 (local address 127.20.178.124:41439)
0929 03:18:00.001174 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0929 03:18:00.001176 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:18:00.004625 (+ 3449us) server_negotiation.cc:407] Connection header received
0929 03:18:00.004781 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:18:00.004782 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:18:00.004795 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:18:00.004872 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:18:00.005556 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:00.006337 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:00.007404 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:00.007766 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:00.007847 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:00.011104 (+ 3257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:18:00.011105 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:18:00.011105 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:18:00.011112 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:18:00.013104 (+ 1992us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:18:00.013436 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:18:00.013437 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:18:00.013437 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:18:00.013492 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:18:00.013696 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:18:00.013697 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:18:00.013698 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:18:00.013811 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:18:00.013913 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0929 03:18:00.014041 (+ 128us) server_negotiation.cc:299] Negotiation successful
0929 03:18:00.014126 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":216,"thread_start_us":49,"threads_started":1}
I20260929 03:18:00.012465 27297 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42490:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 03:18:00.014781 27297 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 03:18:00.015475 27857 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4b1269455e04063b0eb4e2aa46d0684 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d4b1269455e04063b0eb4e2aa46d0684" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d4b1269455e04063b0eb4e2aa46d0684" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 35553 } } peers { permanent_uuid: "82e1ecef80f449029732e3ade3c4a092" member_type: VOTER last_known_addr { host: "127.20.178.125" port: 35183 } attrs { promote: false } } peers { permanent_uuid: "bce5b828f0a947c288a8cb13eb6c6819" member_type: NON_VOTER last_known_addr { host: "127.20.178.124" port: 41439 } attrs { promote: true } } }
I20260929 03:18:00.015578 27857 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4b1269455e04063b0eb4e2aa46d0684 [sys.catalog]: This master's current role is: LEADER
I20260929 03:18:00.016867 27859 raft_consensus.cc:1050] T 00000000000000000000000000000000 P d4b1269455e04063b0eb4e2aa46d0684: attempting to promote NON_VOTER bce5b828f0a947c288a8cb13eb6c6819 to VOTER
I20260929 03:18:00.016985 27859 consensus_queue.cc:237] T 00000000000000000000000000000000 P d4b1269455e04063b0eb4e2aa46d0684 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d4b1269455e04063b0eb4e2aa46d0684" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 35553 } } peers { permanent_uuid: "82e1ecef80f449029732e3ade3c4a092" member_type: VOTER last_known_addr { host: "127.20.178.125" port: 35183 } attrs { promote: false } } peers { permanent_uuid: "bce5b828f0a947c288a8cb13eb6c6819" member_type: VOTER last_known_addr { host: "127.20.178.124" port: 41439 } attrs { promote: false } }
I20260929 03:18:00.017581 28092 raft_consensus.cc:1261] T 00000000000000000000000000000000 P bce5b828f0a947c288a8cb13eb6c6819 [term 1 LEARNER]: Refusing update from remote peer d4b1269455e04063b0eb4e2aa46d0684: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260929 03:18:00.017758 27907 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 82e1ecef80f449029732e3ade3c4a092 [term 1 FOLLOWER]: Refusing update from remote peer d4b1269455e04063b0eb4e2aa46d0684: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260929 03:18:00.017755 27858 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d4b1269455e04063b0eb4e2aa46d0684 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bce5b828f0a947c288a8cb13eb6c6819" member_type: VOTER last_known_addr { host: "127.20.178.124" port: 41439 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20260929 03:18:00.018150 27858 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d4b1269455e04063b0eb4e2aa46d0684 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82e1ecef80f449029732e3ade3c4a092" member_type: VOTER last_known_addr { host: "127.20.178.125" port: 35183 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20260929 03:18:00.018730 27858 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d4b1269455e04063b0eb4e2aa46d0684 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, bce5b828f0a947c288a8cb13eb6c6819 (127.20.178.124) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d4b1269455e04063b0eb4e2aa46d0684" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 35553 } } peers { permanent_uuid: "82e1ecef80f449029732e3ade3c4a092" member_type: VOTER last_known_addr { host: "127.20.178.125" port: 35183 } attrs { promote: false } } peers { permanent_uuid: "bce5b828f0a947c288a8cb13eb6c6819" member_type: VOTER last_known_addr { host: "127.20.178.124" port: 41439 } attrs { promote: false } } }
I20260929 03:18:00.019528 27858 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4b1269455e04063b0eb4e2aa46d0684 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "d4b1269455e04063b0eb4e2aa46d0684" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d4b1269455e04063b0eb4e2aa46d0684" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 35553 } } peers { permanent_uuid: "82e1ecef80f449029732e3ade3c4a092" member_type: VOTER last_known_addr { host: "127.20.178.125" port: 35183 } attrs { promote: false } } peers { permanent_uuid: "bce5b828f0a947c288a8cb13eb6c6819" member_type: VOTER last_known_addr { host: "127.20.178.124" port: 41439 } attrs { promote: false } } }
I20260929 03:18:00.019616 27858 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4b1269455e04063b0eb4e2aa46d0684 [sys.catalog]: This master's current role is: LEADER
I20260929 03:18:00.019672 27858 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4b1269455e04063b0eb4e2aa46d0684 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d4b1269455e04063b0eb4e2aa46d0684" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d4b1269455e04063b0eb4e2aa46d0684" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 35553 } } peers { permanent_uuid: "82e1ecef80f449029732e3ade3c4a092" member_type: VOTER last_known_addr { host: "127.20.178.125" port: 35183 } attrs { promote: false } } peers { permanent_uuid: "bce5b828f0a947c288a8cb13eb6c6819" member_type: VOTER last_known_addr { host: "127.20.178.124" port: 41439 } attrs { promote: false } } }
I20260929 03:18:00.019726 27858 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4b1269455e04063b0eb4e2aa46d0684 [sys.catalog]: This master's current role is: LEADER
I20260929 03:18:00.019765 27858 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4b1269455e04063b0eb4e2aa46d0684 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d4b1269455e04063b0eb4e2aa46d0684" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d4b1269455e04063b0eb4e2aa46d0684" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 35553 } } peers { permanent_uuid: "82e1ecef80f449029732e3ade3c4a092" member_type: VOTER last_known_addr { host: "127.20.178.125" port: 35183 } attrs { promote: false } } peers { permanent_uuid: "bce5b828f0a947c288a8cb13eb6c6819" member_type: VOTER last_known_addr { host: "127.20.178.124" port: 41439 } attrs { promote: false } } }
I20260929 03:18:00.019800 27858 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4b1269455e04063b0eb4e2aa46d0684 [sys.catalog]: This master's current role is: LEADER
I20260929 03:18:00.019961 28549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:00.001762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54218 (local address 127.20.178.125:35183)
0929 03:18:00.004409 (+ 2647us) server_negotiation.cc:206] Beginning negotiation
0929 03:18:00.004411 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:18:00.004419 (+ 8us) server_negotiation.cc:407] Connection header received
0929 03:18:00.004437 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:18:00.004438 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:18:00.004449 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:18:00.004504 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:18:00.013179 (+ 8675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:00.013842 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:00.015046 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:00.015308 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:00.015379 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:00.016917 (+ 1538us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:18:00.016918 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:18:00.016918 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:18:00.016924 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:18:00.018763 (+ 1839us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:18:00.019105 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:18:00.019106 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:18:00.019107 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:18:00.019165 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:18:00.019352 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:18:00.019354 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:18:00.019354 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:18:00.019499 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:18:00.019588 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0929 03:18:00.019763 (+ 175us) server_negotiation.cc:299] Negotiation successful
0929 03:18:00.019861 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2604,"thread_start_us":57,"threads_started":1}
I20260929 03:18:00.020126 27907 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 82e1ecef80f449029732e3ade3c4a092 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, bce5b828f0a947c288a8cb13eb6c6819 (127.20.178.124) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d4b1269455e04063b0eb4e2aa46d0684" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 35553 } } peers { permanent_uuid: "82e1ecef80f449029732e3ade3c4a092" member_type: VOTER last_known_addr { host: "127.20.178.125" port: 35183 } attrs { promote: false } } peers { permanent_uuid: "bce5b828f0a947c288a8cb13eb6c6819" member_type: VOTER last_known_addr { host: "127.20.178.124" port: 41439 } attrs { promote: false } } }
I20260929 03:18:00.020328 28092 raft_consensus.cc:2932] T 00000000000000000000000000000000 P bce5b828f0a947c288a8cb13eb6c6819 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, bce5b828f0a947c288a8cb13eb6c6819 (127.20.178.124) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d4b1269455e04063b0eb4e2aa46d0684" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 35553 } } peers { permanent_uuid: "82e1ecef80f449029732e3ade3c4a092" member_type: VOTER last_known_addr { host: "127.20.178.125" port: 35183 } attrs { promote: false } } peers { permanent_uuid: "bce5b828f0a947c288a8cb13eb6c6819" member_type: VOTER last_known_addr { host: "127.20.178.124" port: 41439 } attrs { promote: false } } }
I20260929 03:18:00.021100 28046 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82e1ecef80f449029732e3ade3c4a092 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d4b1269455e04063b0eb4e2aa46d0684" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d4b1269455e04063b0eb4e2aa46d0684" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 35553 } } peers { permanent_uuid: "82e1ecef80f449029732e3ade3c4a092" member_type: VOTER last_known_addr { host: "127.20.178.125" port: 35183 } attrs { promote: false } } peers { permanent_uuid: "bce5b828f0a947c288a8cb13eb6c6819" member_type: VOTER last_known_addr { host: "127.20.178.124" port: 41439 } attrs { promote: false } } }
I20260929 03:18:00.021193 28046 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82e1ecef80f449029732e3ade3c4a092 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 03:18:00.021273 28121 sys_catalog.cc:455] T 00000000000000000000000000000000 P bce5b828f0a947c288a8cb13eb6c6819 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d4b1269455e04063b0eb4e2aa46d0684" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d4b1269455e04063b0eb4e2aa46d0684" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 35553 } } peers { permanent_uuid: "82e1ecef80f449029732e3ade3c4a092" member_type: VOTER last_known_addr { host: "127.20.178.125" port: 35183 } attrs { promote: false } } peers { permanent_uuid: "bce5b828f0a947c288a8cb13eb6c6819" member_type: VOTER last_known_addr { host: "127.20.178.124" port: 41439 } attrs { promote: false } } }
I20260929 03:18:00.021379 28121 sys_catalog.cc:458] T 00000000000000000000000000000000 P bce5b828f0a947c288a8cb13eb6c6819 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 03:18:00.026885 27856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:00.022557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.66:42401 (local address 127.0.0.1:58420)
0929 03:18:00.022608 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:00.022618 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 03:18:00.022652 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:18:00.023292 (+ 640us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:18:00.023293 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:18:00.023295 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:18:00.023341 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:00.023343 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:00.024444 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:00.024444 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:00.025744 (+ 1300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:00.025747 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:00.026615 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:00.026616 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:00.026685 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:00.026689 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 03:18:00.026738 (+ 49us) client_negotiation.cc:240] Negotiation successful
0929 03:18:00.026778 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26,"mutex_wait_us":78}
I20260929 03:18:00.026952 28561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:00.022965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37010 (local address 127.20.178.67:32833)
0929 03:18:00.023166 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0929 03:18:00.023171 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:18:00.023180 (+ 9us) server_negotiation.cc:407] Connection header received
0929 03:18:00.023216 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:18:00.023218 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:18:00.023232 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:18:00.023432 (+ 200us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:18:00.023485 (+ 53us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:00.024574 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:00.025930 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:00.026620 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:00.026715 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:00.026789 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0929 03:18:00.026842 (+ 53us) server_negotiation.cc:299] Negotiation successful
0929 03:18:00.026879 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":101,"threads_started":1}
I20260929 03:18:00.026961 28559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:00.022653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58420 (local address 127.20.178.66:42401)
0929 03:18:00.023131 (+ 478us) server_negotiation.cc:206] Beginning negotiation
0929 03:18:00.023136 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:18:00.023146 (+ 10us) server_negotiation.cc:407] Connection header received
0929 03:18:00.023186 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:18:00.023188 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:18:00.023205 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:18:00.023295 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:18:00.023411 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:00.024355 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:00.025827 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:00.026526 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:00.026596 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:00.026672 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0929 03:18:00.026843 (+ 171us) server_negotiation.cc:299] Negotiation successful
0929 03:18:00.026887 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":444,"thread_start_us":71,"threads_started":1}
I20260929 03:18:00.026964 28558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:00.022594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.67:32833 (local address 127.0.0.1:37010)
0929 03:18:00.022739 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:00.022749 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 03:18:00.022786 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:18:00.023331 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:18:00.023331 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:18:00.023333 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:18:00.023364 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:00.023366 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:00.024694 (+ 1328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:00.024694 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:00.025840 (+ 1146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:00.025843 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:00.026719 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:00.026720 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:00.026775 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:00.026777 (+ 2us) client_negotiation.cc:769] Sending connection context
0929 03:18:00.026830 (+ 53us) client_negotiation.cc:240] Negotiation successful
0929 03:18:00.026865 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"mutex_wait_us":23,"thread_start_us":88,"threads_started":1}
I20260929 03:18:00.027215 28562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:00.023042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55464 (local address 127.20.178.65:45139)
0929 03:18:00.023222 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0929 03:18:00.023227 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:18:00.023238 (+ 11us) server_negotiation.cc:407] Connection header received
0929 03:18:00.023273 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:18:00.023275 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:18:00.023290 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:18:00.023357 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:18:00.023559 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:00.024683 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:00.025655 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:00.026355 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:00.026426 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:00.026495 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0929 03:18:00.027055 (+ 560us) server_negotiation.cc:299] Negotiation successful
0929 03:18:00.027106 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":117,"threads_started":1}
I20260929 03:18:00.026801 28560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:00.022775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.65:45139 (local address 127.0.0.1:55464)
0929 03:18:00.022898 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:00.022908 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 03:18:00.022948 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:18:00.023425 (+ 477us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:18:00.023426 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:18:00.023428 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:18:00.023479 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:00.023481 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:00.024749 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:00.024749 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:00.025558 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:00.025561 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:00.026533 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:00.026535 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:00.026614 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:00.026618 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 03:18:00.026670 (+ 52us) client_negotiation.cc:240] Negotiation successful
0929 03:18:00.026716 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"mutex_wait_us":1,"thread_start_us":64,"threads_started":1}
I20260929 03:18:00.027669 28468 tablet_service.cc:1467] Processing CreateTablet for tablet 50fd59ad992e41d7994fb36b358d7051 (DEFAULT_TABLE table=test-table [id=522f0fe10fe74fc79c6501f447c983d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:18:00.028020 28468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50fd59ad992e41d7994fb36b358d7051. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:18:00.028055 28195 tablet_service.cc:1467] Processing CreateTablet for tablet 50fd59ad992e41d7994fb36b358d7051 (DEFAULT_TABLE table=test-table [id=522f0fe10fe74fc79c6501f447c983d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:18:00.028018 28332 tablet_service.cc:1467] Processing CreateTablet for tablet 50fd59ad992e41d7994fb36b358d7051 (DEFAULT_TABLE table=test-table [id=522f0fe10fe74fc79c6501f447c983d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:18:00.028349 28195 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50fd59ad992e41d7994fb36b358d7051. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:18:00.028375 28332 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50fd59ad992e41d7994fb36b358d7051. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:18:00.031073 28563 tablet_bootstrap.cc:492] T 50fd59ad992e41d7994fb36b358d7051 P 5064f9a44a034c0dadb4054ac8824780: Bootstrap starting.
I20260929 03:18:00.031457 28564 tablet_bootstrap.cc:492] T 50fd59ad992e41d7994fb36b358d7051 P b2e7ad7ddf214b3596c571f5cd4d3272: Bootstrap starting.
I20260929 03:18:00.031675 28565 tablet_bootstrap.cc:492] T 50fd59ad992e41d7994fb36b358d7051 P e76cae1a85be4646b41cc2ed226fe5c7: Bootstrap starting.
I20260929 03:18:00.032014 28563 tablet_bootstrap.cc:654] T 50fd59ad992e41d7994fb36b358d7051 P 5064f9a44a034c0dadb4054ac8824780: Neither blocks nor log segments found. Creating new log.
I20260929 03:18:00.032044 28564 tablet_bootstrap.cc:654] T 50fd59ad992e41d7994fb36b358d7051 P b2e7ad7ddf214b3596c571f5cd4d3272: Neither blocks nor log segments found. Creating new log.
I20260929 03:18:00.032241 28565 tablet_bootstrap.cc:654] T 50fd59ad992e41d7994fb36b358d7051 P e76cae1a85be4646b41cc2ed226fe5c7: Neither blocks nor log segments found. Creating new log.
I20260929 03:18:00.032352 28564 log.cc:824] T 50fd59ad992e41d7994fb36b358d7051 P b2e7ad7ddf214b3596c571f5cd4d3272: Log is configured to *not* fsync() on all Append() calls
I20260929 03:18:00.032352 28563 log.cc:824] T 50fd59ad992e41d7994fb36b358d7051 P 5064f9a44a034c0dadb4054ac8824780: Log is configured to *not* fsync() on all Append() calls
I20260929 03:18:00.032575 28565 log.cc:824] T 50fd59ad992e41d7994fb36b358d7051 P e76cae1a85be4646b41cc2ed226fe5c7: Log is configured to *not* fsync() on all Append() calls
I20260929 03:18:00.033219 28564 tablet_bootstrap.cc:492] T 50fd59ad992e41d7994fb36b358d7051 P b2e7ad7ddf214b3596c571f5cd4d3272: No bootstrap required, opened a new log
I20260929 03:18:00.033219 28563 tablet_bootstrap.cc:492] T 50fd59ad992e41d7994fb36b358d7051 P 5064f9a44a034c0dadb4054ac8824780: No bootstrap required, opened a new log
I20260929 03:18:00.033290 28563 ts_tablet_manager.cc:1397] T 50fd59ad992e41d7994fb36b358d7051 P 5064f9a44a034c0dadb4054ac8824780: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 03:18:00.033290 28564 ts_tablet_manager.cc:1397] T 50fd59ad992e41d7994fb36b358d7051 P b2e7ad7ddf214b3596c571f5cd4d3272: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 03:18:00.033454 28565 tablet_bootstrap.cc:492] T 50fd59ad992e41d7994fb36b358d7051 P e76cae1a85be4646b41cc2ed226fe5c7: No bootstrap required, opened a new log
I20260929 03:18:00.033553 28565 ts_tablet_manager.cc:1397] T 50fd59ad992e41d7994fb36b358d7051 P e76cae1a85be4646b41cc2ed226fe5c7: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260929 03:18:00.034830 28565 raft_consensus.cc:348] T 50fd59ad992e41d7994fb36b358d7051 P e76cae1a85be4646b41cc2ed226fe5c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5064f9a44a034c0dadb4054ac8824780" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 42401 } } peers { permanent_uuid: "b2e7ad7ddf214b3596c571f5cd4d3272" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 32833 } } peers { permanent_uuid: "e76cae1a85be4646b41cc2ed226fe5c7" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 45139 } }
I20260929 03:18:00.034983 28565 raft_consensus.cc:374] T 50fd59ad992e41d7994fb36b358d7051 P e76cae1a85be4646b41cc2ed226fe5c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:18:00.035027 28565 raft_consensus.cc:729] T 50fd59ad992e41d7994fb36b358d7051 P e76cae1a85be4646b41cc2ed226fe5c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e76cae1a85be4646b41cc2ed226fe5c7, State: Initialized, Role: FOLLOWER
I20260929 03:18:00.035157 28565 consensus_queue.cc:260] T 50fd59ad992e41d7994fb36b358d7051 P e76cae1a85be4646b41cc2ed226fe5c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5064f9a44a034c0dadb4054ac8824780" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 42401 } } peers { permanent_uuid: "b2e7ad7ddf214b3596c571f5cd4d3272" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 32833 } } peers { permanent_uuid: "e76cae1a85be4646b41cc2ed226fe5c7" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 45139 } }
I20260929 03:18:00.035460 28262 heartbeater.cc:503] Master 127.20.178.126:35553 was elected leader, sending a full tablet report...
I20260929 03:18:00.035449 28565 ts_tablet_manager.cc:1428] T 50fd59ad992e41d7994fb36b358d7051 P e76cae1a85be4646b41cc2ed226fe5c7: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260929 03:18:00.035748 28564 raft_consensus.cc:348] T 50fd59ad992e41d7994fb36b358d7051 P b2e7ad7ddf214b3596c571f5cd4d3272 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5064f9a44a034c0dadb4054ac8824780" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 42401 } } peers { permanent_uuid: "b2e7ad7ddf214b3596c571f5cd4d3272" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 32833 } } peers { permanent_uuid: "e76cae1a85be4646b41cc2ed226fe5c7" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 45139 } }
I20260929 03:18:00.035806 28563 raft_consensus.cc:348] T 50fd59ad992e41d7994fb36b358d7051 P 5064f9a44a034c0dadb4054ac8824780 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5064f9a44a034c0dadb4054ac8824780" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 42401 } } peers { permanent_uuid: "b2e7ad7ddf214b3596c571f5cd4d3272" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 32833 } } peers { permanent_uuid: "e76cae1a85be4646b41cc2ed226fe5c7" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 45139 } }
I20260929 03:18:00.035908 28564 raft_consensus.cc:374] T 50fd59ad992e41d7994fb36b358d7051 P b2e7ad7ddf214b3596c571f5cd4d3272 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:18:00.035923 28563 raft_consensus.cc:374] T 50fd59ad992e41d7994fb36b358d7051 P 5064f9a44a034c0dadb4054ac8824780 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:18:00.035955 28564 raft_consensus.cc:729] T 50fd59ad992e41d7994fb36b358d7051 P b2e7ad7ddf214b3596c571f5cd4d3272 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2e7ad7ddf214b3596c571f5cd4d3272, State: Initialized, Role: FOLLOWER
I20260929 03:18:00.035957 28563 raft_consensus.cc:729] T 50fd59ad992e41d7994fb36b358d7051 P 5064f9a44a034c0dadb4054ac8824780 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5064f9a44a034c0dadb4054ac8824780, State: Initialized, Role: FOLLOWER
I20260929 03:18:00.036067 28563 consensus_queue.cc:260] T 50fd59ad992e41d7994fb36b358d7051 P 5064f9a44a034c0dadb4054ac8824780 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5064f9a44a034c0dadb4054ac8824780" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 42401 } } peers { permanent_uuid: "b2e7ad7ddf214b3596c571f5cd4d3272" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 32833 } } peers { permanent_uuid: "e76cae1a85be4646b41cc2ed226fe5c7" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 45139 } }
I20260929 03:18:00.036118 28564 consensus_queue.cc:260] T 50fd59ad992e41d7994fb36b358d7051 P b2e7ad7ddf214b3596c571f5cd4d3272 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5064f9a44a034c0dadb4054ac8824780" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 42401 } } peers { permanent_uuid: "b2e7ad7ddf214b3596c571f5cd4d3272" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 32833 } } peers { permanent_uuid: "e76cae1a85be4646b41cc2ed226fe5c7" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 45139 } }
I20260929 03:18:00.036335 28564 ts_tablet_manager.cc:1428] T 50fd59ad992e41d7994fb36b358d7051 P b2e7ad7ddf214b3596c571f5cd4d3272: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260929 03:18:00.036327 28563 ts_tablet_manager.cc:1428] T 50fd59ad992e41d7994fb36b358d7051 P 5064f9a44a034c0dadb4054ac8824780: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260929 03:18:00.036417 28535 heartbeater.cc:503] Master 127.20.178.126:35553 was elected leader, sending a full tablet report...
I20260929 03:18:00.037840 28399 heartbeater.cc:503] Master 127.20.178.126:35553 was elected leader, sending a full tablet report...
W20260929 03:18:00.108223 28401 tablet.cc:2369] T 50fd59ad992e41d7994fb36b358d7051 P 5064f9a44a034c0dadb4054ac8824780: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 03:18:00.195467 28569 raft_consensus.cc:482] T 50fd59ad992e41d7994fb36b358d7051 P e76cae1a85be4646b41cc2ed226fe5c7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:18:00.195622 28569 raft_consensus.cc:504] T 50fd59ad992e41d7994fb36b358d7051 P e76cae1a85be4646b41cc2ed226fe5c7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5064f9a44a034c0dadb4054ac8824780" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 42401 } } peers { permanent_uuid: "b2e7ad7ddf214b3596c571f5cd4d3272" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 32833 } } peers { permanent_uuid: "e76cae1a85be4646b41cc2ed226fe5c7" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 45139 } }
I20260929 03:18:00.196063 28569 leader_election.cc:288] T 50fd59ad992e41d7994fb36b358d7051 P e76cae1a85be4646b41cc2ed226fe5c7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5064f9a44a034c0dadb4054ac8824780 (127.20.178.66:42401), b2e7ad7ddf214b3596c571f5cd4d3272 (127.20.178.67:32833)
I20260929 03:18:00.199779 28561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:00.196306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.65:59587 (local address 127.20.178.67:32833)
0929 03:18:00.196369 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0929 03:18:00.196372 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:18:00.196390 (+ 18us) server_negotiation.cc:407] Connection header received
0929 03:18:00.196419 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:18:00.196420 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:18:00.196429 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:18:00.196483 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:18:00.196669 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:00.197395 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:00.198833 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:00.199388 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:00.199468 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:00.199476 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0929 03:18:00.199635 (+ 159us) server_negotiation.cc:299] Negotiation successful
0929 03:18:00.199703 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 03:18:00.199819 28265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:00.196226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.67:32833 (local address 127.20.178.65:59587)
0929 03:18:00.196304 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:00.196315 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 03:18:00.196364 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:18:00.196499 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:18:00.196500 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:18:00.196502 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:18:00.196577 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:00.196579 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:00.197481 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:00.197481 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:00.198745 (+ 1264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:00.198748 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:00.199476 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:00.199476 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:00.199553 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:00.199557 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 03:18:00.199617 (+ 60us) client_negotiation.cc:240] Negotiation successful
0929 03:18:00.199664 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52,"mutex_wait_us":17}
I20260929 03:18:00.199869 28559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:00.196297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.65:52351 (local address 127.20.178.66:42401)
0929 03:18:00.196443 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0929 03:18:00.196445 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:18:00.196452 (+ 7us) server_negotiation.cc:407] Connection header received
0929 03:18:00.196476 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:18:00.196477 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:18:00.196486 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:18:00.196539 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:18:00.196662 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:00.197409 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:00.198833 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:00.199406 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:00.199468 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:00.199476 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0929 03:18:00.199750 (+ 274us) server_negotiation.cc:299] Negotiation successful
0929 03:18:00.199797 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122}
I20260929 03:18:00.200194 28488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50fd59ad992e41d7994fb36b358d7051" candidate_uuid: "e76cae1a85be4646b41cc2ed226fe5c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2e7ad7ddf214b3596c571f5cd4d3272" is_pre_election: true
I20260929 03:18:00.199841 28267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:00.196248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.66:42401 (local address 127.20.178.65:52351)
0929 03:18:00.196319 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:00.196328 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 03:18:00.196386 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:18:00.196542 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:18:00.196542 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:18:00.196544 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:18:00.196585 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:00.196586 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:00.197481 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:00.197482 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:00.198745 (+ 1263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:00.198748 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:00.199475 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:00.199476 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:00.199562 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:00.199565 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 03:18:00.199632 (+ 67us) client_negotiation.cc:240] Negotiation successful
0929 03:18:00.199664 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52,"mutex_wait_us":86}
I20260929 03:18:00.200354 28488 raft_consensus.cc:2454] T 50fd59ad992e41d7994fb36b358d7051 P b2e7ad7ddf214b3596c571f5cd4d3272 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e76cae1a85be4646b41cc2ed226fe5c7 in term 0.
I20260929 03:18:00.200541 28352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50fd59ad992e41d7994fb36b358d7051" candidate_uuid: "e76cae1a85be4646b41cc2ed226fe5c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5064f9a44a034c0dadb4054ac8824780" is_pre_election: true
I20260929 03:18:00.200709 28352 raft_consensus.cc:2454] T 50fd59ad992e41d7994fb36b358d7051 P 5064f9a44a034c0dadb4054ac8824780 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e76cae1a85be4646b41cc2ed226fe5c7 in term 0.
I20260929 03:18:00.200920 28148 leader_election.cc:302] T 50fd59ad992e41d7994fb36b358d7051 P e76cae1a85be4646b41cc2ed226fe5c7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5064f9a44a034c0dadb4054ac8824780, e76cae1a85be4646b41cc2ed226fe5c7; no voters:
I20260929 03:18:00.201092 28569 raft_consensus.cc:2781] T 50fd59ad992e41d7994fb36b358d7051 P e76cae1a85be4646b41cc2ed226fe5c7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 03:18:00.201193 28569 raft_consensus.cc:482] T 50fd59ad992e41d7994fb36b358d7051 P e76cae1a85be4646b41cc2ed226fe5c7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 03:18:00.201221 28569 raft_consensus.cc:3037] T 50fd59ad992e41d7994fb36b358d7051 P e76cae1a85be4646b41cc2ed226fe5c7 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:18:00.202056 28569 raft_consensus.cc:504] T 50fd59ad992e41d7994fb36b358d7051 P e76cae1a85be4646b41cc2ed226fe5c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5064f9a44a034c0dadb4054ac8824780" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 42401 } } peers { permanent_uuid: "b2e7ad7ddf214b3596c571f5cd4d3272" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 32833 } } peers { permanent_uuid: "e76cae1a85be4646b41cc2ed226fe5c7" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 45139 } }
I20260929 03:18:00.202242 28569 leader_election.cc:288] T 50fd59ad992e41d7994fb36b358d7051 P e76cae1a85be4646b41cc2ed226fe5c7 [CANDIDATE]: Term 1 election: Requested vote from peers 5064f9a44a034c0dadb4054ac8824780 (127.20.178.66:42401), b2e7ad7ddf214b3596c571f5cd4d3272 (127.20.178.67:32833)
I20260929 03:18:00.202364 28352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50fd59ad992e41d7994fb36b358d7051" candidate_uuid: "e76cae1a85be4646b41cc2ed226fe5c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5064f9a44a034c0dadb4054ac8824780"
I20260929 03:18:00.202383 28488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50fd59ad992e41d7994fb36b358d7051" candidate_uuid: "e76cae1a85be4646b41cc2ed226fe5c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2e7ad7ddf214b3596c571f5cd4d3272"
I20260929 03:18:00.202445 28352 raft_consensus.cc:3037] T 50fd59ad992e41d7994fb36b358d7051 P 5064f9a44a034c0dadb4054ac8824780 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:18:00.202473 28488 raft_consensus.cc:3037] T 50fd59ad992e41d7994fb36b358d7051 P b2e7ad7ddf214b3596c571f5cd4d3272 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:18:00.203105 28488 raft_consensus.cc:2454] T 50fd59ad992e41d7994fb36b358d7051 P b2e7ad7ddf214b3596c571f5cd4d3272 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e76cae1a85be4646b41cc2ed226fe5c7 in term 1.
I20260929 03:18:00.203207 28352 raft_consensus.cc:2454] T 50fd59ad992e41d7994fb36b358d7051 P 5064f9a44a034c0dadb4054ac8824780 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e76cae1a85be4646b41cc2ed226fe5c7 in term 1.
I20260929 03:18:00.203337 28151 leader_election.cc:302] T 50fd59ad992e41d7994fb36b358d7051 P e76cae1a85be4646b41cc2ed226fe5c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b2e7ad7ddf214b3596c571f5cd4d3272, e76cae1a85be4646b41cc2ed226fe5c7; no voters:
W20260929 03:18:00.203486 28264 tablet.cc:2369] T 50fd59ad992e41d7994fb36b358d7051 P e76cae1a85be4646b41cc2ed226fe5c7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 03:18:00.203727 28569 raft_consensus.cc:2781] T 50fd59ad992e41d7994fb36b358d7051 P e76cae1a85be4646b41cc2ed226fe5c7 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:18:00.203872 28569 raft_consensus.cc:686] T 50fd59ad992e41d7994fb36b358d7051 P e76cae1a85be4646b41cc2ed226fe5c7 [term 1 LEADER]: Becoming Leader. State: Replica: e76cae1a85be4646b41cc2ed226fe5c7, State: Running, Role: LEADER
I20260929 03:18:00.203967 28569 consensus_queue.cc:237] T 50fd59ad992e41d7994fb36b358d7051 P e76cae1a85be4646b41cc2ed226fe5c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5064f9a44a034c0dadb4054ac8824780" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 42401 } } peers { permanent_uuid: "b2e7ad7ddf214b3596c571f5cd4d3272" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 32833 } } peers { permanent_uuid: "e76cae1a85be4646b41cc2ed226fe5c7" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 45139 } }
I20260929 03:18:00.204723 27298 catalog_manager.cc:5495] T 50fd59ad992e41d7994fb36b358d7051 P e76cae1a85be4646b41cc2ed226fe5c7 reported cstate change: term changed from 0 to 1, leader changed from <none> to e76cae1a85be4646b41cc2ed226fe5c7 (127.20.178.65). New cstate: current_term: 1 leader_uuid: "e76cae1a85be4646b41cc2ed226fe5c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5064f9a44a034c0dadb4054ac8824780" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 42401 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b2e7ad7ddf214b3596c571f5cd4d3272" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 32833 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e76cae1a85be4646b41cc2ed226fe5c7" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 45139 } health_report { overall_health: HEALTHY } } }
W20260929 03:18:00.245738 28537 tablet.cc:2369] T 50fd59ad992e41d7994fb36b358d7051 P b2e7ad7ddf214b3596c571f5cd4d3272: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 03:18:00.250698 28562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:00.248249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55470 (local address 127.20.178.65:45139)
0929 03:18:00.248317 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0929 03:18:00.248319 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:18:00.248329 (+ 10us) server_negotiation.cc:407] Connection header received
0929 03:18:00.248399 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:18:00.248399 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:18:00.248418 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:18:00.248493 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:18:00.248652 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:00.249123 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:00.249915 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:00.250108 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:00.250183 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:00.250343 (+ 160us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:18:00.250450 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:18:00.250545 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0929 03:18:00.250564 (+ 19us) server_negotiation.cc:299] Negotiation successful
0929 03:18:00.250583 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260929 03:18:00.252365 28352 raft_consensus.cc:1261] T 50fd59ad992e41d7994fb36b358d7051 P 5064f9a44a034c0dadb4054ac8824780 [term 1 FOLLOWER]: Refusing update from remote peer e76cae1a85be4646b41cc2ed226fe5c7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:18:00.252405 28488 raft_consensus.cc:1261] T 50fd59ad992e41d7994fb36b358d7051 P b2e7ad7ddf214b3596c571f5cd4d3272 [term 1 FOLLOWER]: Refusing update from remote peer e76cae1a85be4646b41cc2ed226fe5c7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:18:00.252758 28572 consensus_queue.cc:1035] T 50fd59ad992e41d7994fb36b358d7051 P e76cae1a85be4646b41cc2ed226fe5c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5064f9a44a034c0dadb4054ac8824780" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 42401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:18:00.252911 28569 consensus_queue.cc:1035] T 50fd59ad992e41d7994fb36b358d7051 P e76cae1a85be4646b41cc2ed226fe5c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2e7ad7ddf214b3596c571f5cd4d3272" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 32833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:18:00.255179 28575 mvcc.cc:205] Tried to move back new op lower bound from 7334510101511139328 to 7334510101315969024. Current Snapshot: MvccSnapshot[applied={T|T < 7334510101511139328}]
I20260929 03:18:00.255360 28577 mvcc.cc:205] Tried to move back new op lower bound from 7334510101511139328 to 7334510101315969024. Current Snapshot: MvccSnapshot[applied={T|T < 7334510101511139328}]
I20260929 03:18:00.258140 27298 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42490:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:18:00.258244 27298 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42490:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:18:00.259706 27298 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d4b1269455e04063b0eb4e2aa46d0684: Sending DeleteTablet for 3 replicas of tablet 50fd59ad992e41d7994fb36b358d7051
I20260929 03:18:00.260016 28468 tablet_service.cc:1514] Processing DeleteTablet for tablet 50fd59ad992e41d7994fb36b358d7051 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:18:00 UTC) from {username='oryx', principal='oryx/127.20.178.126@KRBTEST.COM'} at 127.0.0.1:37010
I20260929 03:18:00.260059 28195 tablet_service.cc:1514] Processing DeleteTablet for tablet 50fd59ad992e41d7994fb36b358d7051 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:18:00 UTC) from {username='oryx', principal='oryx/127.20.178.126@KRBTEST.COM'} at 127.0.0.1:55464
I20260929 03:18:00.260231 28332 tablet_service.cc:1514] Processing DeleteTablet for tablet 50fd59ad992e41d7994fb36b358d7051 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:18:00 UTC) from {username='oryx', principal='oryx/127.20.178.126@KRBTEST.COM'} at 127.0.0.1:58420
I20260929 03:18:00.260474 28586 tablet_replica.cc:330] T 50fd59ad992e41d7994fb36b358d7051 P 5064f9a44a034c0dadb4054ac8824780: stopping tablet replica
I20260929 03:18:00.260542 28584 tablet_replica.cc:330] T 50fd59ad992e41d7994fb36b358d7051 P b2e7ad7ddf214b3596c571f5cd4d3272: stopping tablet replica
I20260929 03:18:00.260578 28586 raft_consensus.cc:2229] T 50fd59ad992e41d7994fb36b358d7051 P 5064f9a44a034c0dadb4054ac8824780 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:18:00.260655 28584 raft_consensus.cc:2229] T 50fd59ad992e41d7994fb36b358d7051 P b2e7ad7ddf214b3596c571f5cd4d3272 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:18:00.260711 28586 raft_consensus.cc:2258] T 50fd59ad992e41d7994fb36b358d7051 P 5064f9a44a034c0dadb4054ac8824780 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:18:00.261294 28584 raft_consensus.cc:2258] T 50fd59ad992e41d7994fb36b358d7051 P b2e7ad7ddf214b3596c571f5cd4d3272 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:18:00.261322 21193 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuoaHbX/build/release/bin/kudu with pid 27266
I20260929 03:18:00.261381 28586 ts_tablet_manager.cc:1905] T 50fd59ad992e41d7994fb36b358d7051 P 5064f9a44a034c0dadb4054ac8824780: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:18:00.263797 28586 ts_tablet_manager.cc:1918] T 50fd59ad992e41d7994fb36b358d7051 P 5064f9a44a034c0dadb4054ac8824780: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:18:00.263870 28586 log.cc:1196] T 50fd59ad992e41d7994fb36b358d7051 P 5064f9a44a034c0dadb4054ac8824780: Deleting WAL directory at /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/minicluster-data/ts-1/wal/wals/50fd59ad992e41d7994fb36b358d7051
I20260929 03:18:00.264112 28586 ts_tablet_manager.cc:1939] T 50fd59ad992e41d7994fb36b358d7051 P 5064f9a44a034c0dadb4054ac8824780: Deleting consensus metadata
I20260929 03:18:00.265064 28584 ts_tablet_manager.cc:1905] T 50fd59ad992e41d7994fb36b358d7051 P b2e7ad7ddf214b3596c571f5cd4d3272: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:18:00.266677 28584 ts_tablet_manager.cc:1918] T 50fd59ad992e41d7994fb36b358d7051 P b2e7ad7ddf214b3596c571f5cd4d3272: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:18:00.266750 28584 log.cc:1196] T 50fd59ad992e41d7994fb36b358d7051 P b2e7ad7ddf214b3596c571f5cd4d3272: Deleting WAL directory at /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/minicluster-data/ts-2/wal/wals/50fd59ad992e41d7994fb36b358d7051
I20260929 03:18:00.267004 28584 ts_tablet_manager.cc:1939] T 50fd59ad992e41d7994fb36b358d7051 P b2e7ad7ddf214b3596c571f5cd4d3272: Deleting consensus metadata
I20260929 03:18:00.267453 28585 tablet_replica.cc:330] T 50fd59ad992e41d7994fb36b358d7051 P e76cae1a85be4646b41cc2ed226fe5c7: stopping tablet replica
W20260929 03:18:00.267493 28286 connection.cc:535] server connection from 127.0.0.1:58420 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260929 03:18:00.267578 28585 raft_consensus.cc:2229] T 50fd59ad992e41d7994fb36b358d7051 P e76cae1a85be4646b41cc2ed226fe5c7 [term 1 LEADER]: Raft consensus shutting down.
I20260929 03:18:00.267689 28585 raft_consensus.cc:2258] T 50fd59ad992e41d7994fb36b358d7051 P e76cae1a85be4646b41cc2ed226fe5c7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:18:00.268261 28585 ts_tablet_manager.cc:1905] T 50fd59ad992e41d7994fb36b358d7051 P e76cae1a85be4646b41cc2ed226fe5c7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:18:00.269838 28585 ts_tablet_manager.cc:1918] T 50fd59ad992e41d7994fb36b358d7051 P e76cae1a85be4646b41cc2ed226fe5c7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:18:00.269913 28585 log.cc:1196] T 50fd59ad992e41d7994fb36b358d7051 P e76cae1a85be4646b41cc2ed226fe5c7: Deleting WAL directory at /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/minicluster-data/ts-0/wal/wals/50fd59ad992e41d7994fb36b358d7051
I20260929 03:18:00.270177 28585 ts_tablet_manager.cc:1939] T 50fd59ad992e41d7994fb36b358d7051 P e76cae1a85be4646b41cc2ed226fe5c7: Deleting consensus metadata
W20260929 03:18:00.270488 28151 connection.cc:441] server connection from 127.0.0.1:55464 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55464 (request call id 1) could send its response
I20260929 03:18:00.277040 28545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:00.260912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44080 (local address 127.20.178.124:41439)
0929 03:18:00.260958 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0929 03:18:00.260960 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:18:00.264781 (+ 3821us) server_negotiation.cc:407] Connection header received
0929 03:18:00.267380 (+ 2599us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:18:00.267381 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:18:00.267397 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:18:00.267461 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:18:00.268712 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:00.269461 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:00.270801 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:00.271126 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:00.271197 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:00.272892 (+ 1695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:18:00.272894 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:18:00.272894 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:18:00.272901 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:18:00.275443 (+ 2542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:18:00.275946 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:18:00.275947 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:18:00.275947 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:18:00.276003 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:18:00.276197 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:18:00.276197 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:18:00.276198 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:18:00.276319 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:18:00.276397 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0929 03:18:00.276587 (+ 190us) server_negotiation.cc:299] Negotiation successful
0929 03:18:00.276932 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260929 03:18:00.278505 28268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:00.269229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44086 (local address 127.20.178.124:41439)
0929 03:18:00.269269 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0929 03:18:00.269270 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 03:18:00.269699 (+ 429us) server_negotiation.cc:407] Connection header received
0929 03:18:00.269787 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:18:00.269788 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:18:00.269797 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:18:00.269826 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:18:00.270854 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:00.271540 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:00.272787 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:00.273063 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:00.273097 (+ 34us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:00.274400 (+ 1303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:18:00.274401 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:18:00.274401 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:18:00.274407 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:18:00.276745 (+ 2338us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:18:00.277515 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:18:00.277516 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:18:00.277517 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:18:00.277570 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:18:00.277771 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:18:00.277772 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:18:00.277772 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:18:00.277899 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:18:00.277972 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 03:18:00.278319 (+ 347us) server_negotiation.cc:299] Negotiation successful
0929 03:18:00.278389 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260929 03:18:00.279106 28549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:00.271633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54234 (local address 127.20.178.125:35183)
0929 03:18:00.271706 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0929 03:18:00.271708 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:18:00.271717 (+ 9us) server_negotiation.cc:407] Connection header received
0929 03:18:00.271738 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:18:00.271738 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:18:00.271750 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:18:00.271811 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:18:00.272456 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:00.273198 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:00.274281 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:00.274615 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:00.274657 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:00.275998 (+ 1341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:18:00.275999 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:18:00.275999 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:18:00.276006 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:18:00.277772 (+ 1766us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:18:00.278157 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:18:00.278158 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:18:00.278159 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:18:00.278216 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:18:00.278468 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:18:00.278469 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:18:00.278469 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:18:00.278622 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:18:00.278712 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0929 03:18:00.278903 (+ 191us) server_negotiation.cc:299] Negotiation successful
0929 03:18:00.278984 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 03:18:01.257454 28595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:01.257136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:35553 (local address 127.20.178.66:36789)
0929 03:18:01.257279 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:01.257368 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.178.126:35553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":96,"threads_started":1}
I20260929 03:18:01.257790 28596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:01.257589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:35553 (local address 127.20.178.67:56195)
0929 03:18:01.257685 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:01.257749 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.178.126:35553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81,"thread_start_us":47,"threads_started":1}
W20260929 03:18:01.257867 28399 heartbeater.cc:644] Failed to heartbeat to 127.20.178.126:35553 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.20.178.126:35553: connect: Connection refused (error 111)
W20260929 03:18:01.257965 28535 heartbeater.cc:644] Failed to heartbeat to 127.20.178.126:35553 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.20.178.126:35553: connect: Connection refused (error 111)
I20260929 03:18:01.258471 28597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:01.258287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:35553 (local address 127.20.178.65:50273)
0929 03:18:01.258382 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:01.258437 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.178.126:35553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81,"thread_start_us":45,"threads_started":1}
W20260929 03:18:01.258630 28262 heartbeater.cc:644] Failed to heartbeat to 127.20.178.126:35553 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.20.178.126:35553: connect: Connection refused (error 111)
I20260929 03:18:01.760349 28598 raft_consensus.cc:482] T 00000000000000000000000000000000 P bce5b828f0a947c288a8cb13eb6c6819 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d4b1269455e04063b0eb4e2aa46d0684)
I20260929 03:18:01.760417 28599 raft_consensus.cc:482] T 00000000000000000000000000000000 P 82e1ecef80f449029732e3ade3c4a092 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d4b1269455e04063b0eb4e2aa46d0684)
I20260929 03:18:01.760538 28598 raft_consensus.cc:504] T 00000000000000000000000000000000 P bce5b828f0a947c288a8cb13eb6c6819 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d4b1269455e04063b0eb4e2aa46d0684" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 35553 } } peers { permanent_uuid: "82e1ecef80f449029732e3ade3c4a092" member_type: VOTER last_known_addr { host: "127.20.178.125" port: 35183 } attrs { promote: false } } peers { permanent_uuid: "bce5b828f0a947c288a8cb13eb6c6819" member_type: VOTER last_known_addr { host: "127.20.178.124" port: 41439 } attrs { promote: false } }
I20260929 03:18:01.760537 28599 raft_consensus.cc:504] T 00000000000000000000000000000000 P 82e1ecef80f449029732e3ade3c4a092 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d4b1269455e04063b0eb4e2aa46d0684" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 35553 } } peers { permanent_uuid: "82e1ecef80f449029732e3ade3c4a092" member_type: VOTER last_known_addr { host: "127.20.178.125" port: 35183 } attrs { promote: false } } peers { permanent_uuid: "bce5b828f0a947c288a8cb13eb6c6819" member_type: VOTER last_known_addr { host: "127.20.178.124" port: 41439 } attrs { promote: false } }
I20260929 03:18:01.760836 28598 leader_election.cc:288] T 00000000000000000000000000000000 P bce5b828f0a947c288a8cb13eb6c6819 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d4b1269455e04063b0eb4e2aa46d0684 (127.20.178.126:35553), 82e1ecef80f449029732e3ade3c4a092 (127.20.178.125:35183)
I20260929 03:18:01.761086 28599 leader_election.cc:288] T 00000000000000000000000000000000 P 82e1ecef80f449029732e3ade3c4a092 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d4b1269455e04063b0eb4e2aa46d0684 (127.20.178.126:35553), bce5b828f0a947c288a8cb13eb6c6819 (127.20.178.124:41439)
I20260929 03:18:01.761215 28600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:01.760930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:35553 (local address 127.0.0.1:42974)
0929 03:18:01.761124 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:01.761172 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.178.126:35553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":48,"threads_started":1}
W20260929 03:18:01.761329 28067 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.178.126:35553: connect: Connection refused (error 111)
I20260929 03:18:01.761340 28601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:01.760929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:35553 (local address 127.0.0.1:42976)
0929 03:18:01.761055 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:01.761304 (+ 249us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.178.126:35553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":45,"threads_started":1}
W20260929 03:18:01.761516 27882 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.178.126:35553: connect: Connection refused (error 111)
I20260929 03:18:01.761869 28600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:01.761783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:35553 (local address 127.0.0.1:43000)
0929 03:18:01.761817 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:01.761843 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.178.126:35553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
W20260929 03:18:01.761945 28067 leader_election.cc:334] T 00000000000000000000000000000000 P bce5b828f0a947c288a8cb13eb6c6819 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d4b1269455e04063b0eb4e2aa46d0684 (127.20.178.126:35553): Network error: Client connection negotiation failed: client connection to 127.20.178.126:35553: connect: Connection refused (error 111)
I20260929 03:18:01.761970 28604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:01.761862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:35553 (local address 127.0.0.1:43002)
0929 03:18:01.761900 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:01.761937 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.178.126:35553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260929 03:18:01.762036 27882 leader_election.cc:334] T 00000000000000000000000000000000 P 82e1ecef80f449029732e3ade3c4a092 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d4b1269455e04063b0eb4e2aa46d0684 (127.20.178.126:35553): Network error: Client connection negotiation failed: client connection to 127.20.178.126:35553: connect: Connection refused (error 111)
I20260929 03:18:01.764279 28601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:01.761198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.124:41439 (local address 127.0.0.1:44088)
0929 03:18:01.761405 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:01.761414 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 03:18:01.761461 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:18:01.761832 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:18:01.761833 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:18:01.761836 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:18:01.761885 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:01.761887 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:01.762655 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:01.762656 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:01.763511 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:01.763514 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:01.764072 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:01.764073 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:01.764132 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:01.764137 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 03:18:01.764191 (+ 54us) client_negotiation.cc:240] Negotiation successful
0929 03:18:01.764233 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":46,"threads_started":1}
I20260929 03:18:01.764456 28602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:01.761002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54244 (local address 127.20.178.125:35183)
0929 03:18:01.761071 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0929 03:18:01.761073 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:18:01.761361 (+ 288us) server_negotiation.cc:407] Connection header received
0929 03:18:01.761558 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:18:01.761559 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:18:01.761568 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:18:01.761682 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:18:01.761897 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:01.762733 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:01.763702 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:01.764072 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:01.764164 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:01.764173 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0929 03:18:01.764337 (+ 164us) server_negotiation.cc:299] Negotiation successful
0929 03:18:01.764381 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50,"thread_start_us":32,"threads_started":1}
I20260929 03:18:01.764336 28605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:01.761382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44088 (local address 127.20.178.124:41439)
0929 03:18:01.761531 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0929 03:18:01.761533 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:18:01.761541 (+ 8us) server_negotiation.cc:407] Connection header received
0929 03:18:01.761565 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:18:01.761566 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:18:01.761576 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:18:01.761652 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:18:01.761969 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:01.762543 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:01.763601 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:01.763996 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:01.764050 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:01.764057 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0929 03:18:01.764215 (+ 158us) server_negotiation.cc:299] Negotiation successful
0929 03:18:01.764255 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":55,"threads_started":1}
I20260929 03:18:01.764565 28603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:01.760994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.125:35183 (local address 127.0.0.1:54244)
0929 03:18:01.761211 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:01.761223 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 03:18:01.761478 (+ 255us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:18:01.761732 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:18:01.761733 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:18:01.761734 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:18:01.761812 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:01.761814 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:01.762822 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:01.762823 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:01.763619 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:01.763620 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:01.764201 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:01.764202 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:01.764258 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:01.764261 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 03:18:01.764319 (+ 58us) client_negotiation.cc:240] Negotiation successful
0929 03:18:01.764353 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":67,"threads_started":1}
I20260929 03:18:01.764797 28092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "82e1ecef80f449029732e3ade3c4a092" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "bce5b828f0a947c288a8cb13eb6c6819" is_pre_election: true
I20260929 03:18:01.764846 27907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bce5b828f0a947c288a8cb13eb6c6819" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "82e1ecef80f449029732e3ade3c4a092" is_pre_election: true
I20260929 03:18:01.764902 28092 raft_consensus.cc:2454] T 00000000000000000000000000000000 P bce5b828f0a947c288a8cb13eb6c6819 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82e1ecef80f449029732e3ade3c4a092 in term 1.
I20260929 03:18:01.764938 27907 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 82e1ecef80f449029732e3ade3c4a092 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bce5b828f0a947c288a8cb13eb6c6819 in term 1.
I20260929 03:18:01.765064 27884 leader_election.cc:302] T 00000000000000000000000000000000 P 82e1ecef80f449029732e3ade3c4a092 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 82e1ecef80f449029732e3ade3c4a092, bce5b828f0a947c288a8cb13eb6c6819; no voters: d4b1269455e04063b0eb4e2aa46d0684
I20260929 03:18:01.765116 28066 leader_election.cc:302] T 00000000000000000000000000000000 P bce5b828f0a947c288a8cb13eb6c6819 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 82e1ecef80f449029732e3ade3c4a092, bce5b828f0a947c288a8cb13eb6c6819; no voters: d4b1269455e04063b0eb4e2aa46d0684
I20260929 03:18:01.765205 28599 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 82e1ecef80f449029732e3ade3c4a092 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260929 03:18:01.765219 28598 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bce5b828f0a947c288a8cb13eb6c6819 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260929 03:18:01.765250 28599 raft_consensus.cc:482] T 00000000000000000000000000000000 P 82e1ecef80f449029732e3ade3c4a092 [term 1 FOLLOWER]: Starting leader election (detected failure of leader d4b1269455e04063b0eb4e2aa46d0684)
I20260929 03:18:01.765251 28598 raft_consensus.cc:482] T 00000000000000000000000000000000 P bce5b828f0a947c288a8cb13eb6c6819 [term 1 FOLLOWER]: Starting leader election (detected failure of leader d4b1269455e04063b0eb4e2aa46d0684)
I20260929 03:18:01.765273 28599 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 82e1ecef80f449029732e3ade3c4a092 [term 1 FOLLOWER]: Advancing to term 2
I20260929 03:18:01.765282 28598 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bce5b828f0a947c288a8cb13eb6c6819 [term 1 FOLLOWER]: Advancing to term 2
I20260929 03:18:01.773551 28599 raft_consensus.cc:504] T 00000000000000000000000000000000 P 82e1ecef80f449029732e3ade3c4a092 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d4b1269455e04063b0eb4e2aa46d0684" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 35553 } } peers { permanent_uuid: "82e1ecef80f449029732e3ade3c4a092" member_type: VOTER last_known_addr { host: "127.20.178.125" port: 35183 } attrs { promote: false } } peers { permanent_uuid: "bce5b828f0a947c288a8cb13eb6c6819" member_type: VOTER last_known_addr { host: "127.20.178.124" port: 41439 } attrs { promote: false } }
I20260929 03:18:01.773551 28598 raft_consensus.cc:504] T 00000000000000000000000000000000 P bce5b828f0a947c288a8cb13eb6c6819 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d4b1269455e04063b0eb4e2aa46d0684" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 35553 } } peers { permanent_uuid: "82e1ecef80f449029732e3ade3c4a092" member_type: VOTER last_known_addr { host: "127.20.178.125" port: 35183 } attrs { promote: false } } peers { permanent_uuid: "bce5b828f0a947c288a8cb13eb6c6819" member_type: VOTER last_known_addr { host: "127.20.178.124" port: 41439 } attrs { promote: false } }
I20260929 03:18:01.773744 28599 leader_election.cc:288] T 00000000000000000000000000000000 P 82e1ecef80f449029732e3ade3c4a092 [CANDIDATE]: Term 2 election: Requested vote from peers d4b1269455e04063b0eb4e2aa46d0684 (127.20.178.126:35553), bce5b828f0a947c288a8cb13eb6c6819 (127.20.178.124:41439)
I20260929 03:18:01.773741 28598 leader_election.cc:288] T 00000000000000000000000000000000 P bce5b828f0a947c288a8cb13eb6c6819 [CANDIDATE]: Term 2 election: Requested vote from peers d4b1269455e04063b0eb4e2aa46d0684 (127.20.178.126:35553), 82e1ecef80f449029732e3ade3c4a092 (127.20.178.125:35183)
I20260929 03:18:01.773980 28092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "82e1ecef80f449029732e3ade3c4a092" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "bce5b828f0a947c288a8cb13eb6c6819"
I20260929 03:18:01.774080 28092 raft_consensus.cc:2379] T 00000000000000000000000000000000 P bce5b828f0a947c288a8cb13eb6c6819 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 82e1ecef80f449029732e3ade3c4a092 in current term 2: Already voted for candidate bce5b828f0a947c288a8cb13eb6c6819 in this term.
I20260929 03:18:01.774102 27907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bce5b828f0a947c288a8cb13eb6c6819" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "82e1ecef80f449029732e3ade3c4a092"
I20260929 03:18:01.774163 28601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:01.774017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:35553 (local address 127.0.0.1:43012)
0929 03:18:01.774104 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:01.774131 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.178.126:35553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260929 03:18:01.774205 27907 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 82e1ecef80f449029732e3ade3c4a092 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate bce5b828f0a947c288a8cb13eb6c6819 in current term 2: Already voted for candidate 82e1ecef80f449029732e3ade3c4a092 in this term.
I20260929 03:18:01.774410 28603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:01.773877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:35553 (local address 127.0.0.1:43010)
0929 03:18:01.774030 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:01.774375 (+ 345us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.178.126:35553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136}
I20260929 03:18:01.774528 28601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:01.774406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:35553 (local address 127.0.0.1:43020)
0929 03:18:01.774454 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:01.774481 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.178.126:35553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260929 03:18:01.774600 27882 leader_election.cc:334] T 00000000000000000000000000000000 P 82e1ecef80f449029732e3ade3c4a092 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer d4b1269455e04063b0eb4e2aa46d0684 (127.20.178.126:35553): Network error: Client connection negotiation failed: client connection to 127.20.178.126:35553: connect: Connection refused (error 111)
I20260929 03:18:01.774644 27882 leader_election.cc:302] T 00000000000000000000000000000000 P 82e1ecef80f449029732e3ade3c4a092 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82e1ecef80f449029732e3ade3c4a092; no voters: bce5b828f0a947c288a8cb13eb6c6819, d4b1269455e04063b0eb4e2aa46d0684
I20260929 03:18:01.774729 28599 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 82e1ecef80f449029732e3ade3c4a092 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260929 03:18:01.774857 28603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:01.774733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:35553 (local address 127.0.0.1:43034)
0929 03:18:01.774805 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:01.774831 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.178.126:35553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
W20260929 03:18:01.774942 28067 leader_election.cc:334] T 00000000000000000000000000000000 P bce5b828f0a947c288a8cb13eb6c6819 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer d4b1269455e04063b0eb4e2aa46d0684 (127.20.178.126:35553): Network error: Client connection negotiation failed: client connection to 127.20.178.126:35553: connect: Connection refused (error 111)
I20260929 03:18:01.775005 28067 leader_election.cc:302] T 00000000000000000000000000000000 P bce5b828f0a947c288a8cb13eb6c6819 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bce5b828f0a947c288a8cb13eb6c6819; no voters: 82e1ecef80f449029732e3ade3c4a092, d4b1269455e04063b0eb4e2aa46d0684
I20260929 03:18:01.775058 28598 raft_consensus.cc:2726] T 00000000000000000000000000000000 P bce5b828f0a947c288a8cb13eb6c6819 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260929 03:18:02.258451 28608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:02.258205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:35553 (local address 127.20.178.66:54113)
0929 03:18:02.258353 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:02.258393 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.178.126:35553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":61,"threads_started":1}
I20260929 03:18:02.258579 28609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:02.258245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:35553 (local address 127.20.178.67:60631)
0929 03:18:02.258493 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:02.258532 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.178.126:35553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":72,"threads_started":1}
I20260929 03:18:02.259056 28610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:02.258887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:35553 (local address 127.20.178.65:42481)
0929 03:18:02.258989 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:02.259021 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.178.126:35553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87,"thread_start_us":43,"threads_started":1}
I20260929 03:18:03.259127 28611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:03.258905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:35553 (local address 127.20.178.66:39965)
0929 03:18:03.259027 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:03.259077 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.178.126:35553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":70,"threads_started":1}
I20260929 03:18:03.259126 28612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:03.258951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:35553 (local address 127.20.178.67:32781)
0929 03:18:03.259036 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:03.259073 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.178.126:35553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68,"thread_start_us":38,"threads_started":1}
I20260929 03:18:03.259589 28613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:03.259439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:35553 (local address 127.20.178.65:36765)
0929 03:18:03.259536 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:03.259558 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.178.126:35553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87,"thread_start_us":57,"threads_started":1}
I20260929 03:18:03.614156 28614 raft_consensus.cc:482] T 00000000000000000000000000000000 P bce5b828f0a947c288a8cb13eb6c6819 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:18:03.614274 28614 raft_consensus.cc:504] T 00000000000000000000000000000000 P bce5b828f0a947c288a8cb13eb6c6819 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d4b1269455e04063b0eb4e2aa46d0684" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 35553 } } peers { permanent_uuid: "82e1ecef80f449029732e3ade3c4a092" member_type: VOTER last_known_addr { host: "127.20.178.125" port: 35183 } attrs { promote: false } } peers { permanent_uuid: "bce5b828f0a947c288a8cb13eb6c6819" member_type: VOTER last_known_addr { host: "127.20.178.124" port: 41439 } attrs { promote: false } }
I20260929 03:18:03.614473 28614 leader_election.cc:288] T 00000000000000000000000000000000 P bce5b828f0a947c288a8cb13eb6c6819 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d4b1269455e04063b0eb4e2aa46d0684 (127.20.178.126:35553), 82e1ecef80f449029732e3ade3c4a092 (127.20.178.125:35183)
I20260929 03:18:03.614713 27907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bce5b828f0a947c288a8cb13eb6c6819" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "82e1ecef80f449029732e3ade3c4a092" is_pre_election: true
I20260929 03:18:03.614833 28615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:03.614603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:35553 (local address 127.0.0.1:43254)
0929 03:18:03.614740 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:03.614796 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.178.126:35553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":72,"threads_started":1}
I20260929 03:18:03.614830 27907 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 82e1ecef80f449029732e3ade3c4a092 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bce5b828f0a947c288a8cb13eb6c6819 in term 2.
I20260929 03:18:03.614998 28066 leader_election.cc:302] T 00000000000000000000000000000000 P bce5b828f0a947c288a8cb13eb6c6819 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82e1ecef80f449029732e3ade3c4a092, bce5b828f0a947c288a8cb13eb6c6819; no voters:
I20260929 03:18:03.615109 28614 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bce5b828f0a947c288a8cb13eb6c6819 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260929 03:18:03.615149 28614 raft_consensus.cc:482] T 00000000000000000000000000000000 P bce5b828f0a947c288a8cb13eb6c6819 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 03:18:03.615211 28614 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bce5b828f0a947c288a8cb13eb6c6819 [term 2 FOLLOWER]: Advancing to term 3
I20260929 03:18:03.615283 28615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:03.615153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:35553 (local address 127.0.0.1:43268)
0929 03:18:03.615203 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:03.615234 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.178.126:35553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260929 03:18:03.615389 28067 leader_election.cc:334] T 00000000000000000000000000000000 P bce5b828f0a947c288a8cb13eb6c6819 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d4b1269455e04063b0eb4e2aa46d0684 (127.20.178.126:35553): Network error: Client connection negotiation failed: client connection to 127.20.178.126:35553: connect: Connection refused (error 111)
I20260929 03:18:03.615954 28614 raft_consensus.cc:504] T 00000000000000000000000000000000 P bce5b828f0a947c288a8cb13eb6c6819 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d4b1269455e04063b0eb4e2aa46d0684" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 35553 } } peers { permanent_uuid: "82e1ecef80f449029732e3ade3c4a092" member_type: VOTER last_known_addr { host: "127.20.178.125" port: 35183 } attrs { promote: false } } peers { permanent_uuid: "bce5b828f0a947c288a8cb13eb6c6819" member_type: VOTER last_known_addr { host: "127.20.178.124" port: 41439 } attrs { promote: false } }
I20260929 03:18:03.616089 28614 leader_election.cc:288] T 00000000000000000000000000000000 P bce5b828f0a947c288a8cb13eb6c6819 [CANDIDATE]: Term 3 election: Requested vote from peers d4b1269455e04063b0eb4e2aa46d0684 (127.20.178.126:35553), 82e1ecef80f449029732e3ade3c4a092 (127.20.178.125:35183)
I20260929 03:18:03.616237 28615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:03.616148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:35553 (local address 127.0.0.1:43284)
0929 03:18:03.616183 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:03.616215 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.178.126:35553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
I20260929 03:18:03.616225 27907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bce5b828f0a947c288a8cb13eb6c6819" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "82e1ecef80f449029732e3ade3c4a092"
I20260929 03:18:03.616298 27907 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 82e1ecef80f449029732e3ade3c4a092 [term 2 FOLLOWER]: Advancing to term 3
I20260929 03:18:03.616552 28615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:03.616453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:35553 (local address 127.0.0.1:43292)
0929 03:18:03.616505 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:03.616533 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.178.126:35553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260929 03:18:03.616621 28067 leader_election.cc:334] T 00000000000000000000000000000000 P bce5b828f0a947c288a8cb13eb6c6819 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer d4b1269455e04063b0eb4e2aa46d0684 (127.20.178.126:35553): Network error: Client connection negotiation failed: client connection to 127.20.178.126:35553: connect: Connection refused (error 111)
I20260929 03:18:03.617074 27907 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 82e1ecef80f449029732e3ade3c4a092 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bce5b828f0a947c288a8cb13eb6c6819 in term 3.
I20260929 03:18:03.617252 28066 leader_election.cc:302] T 00000000000000000000000000000000 P bce5b828f0a947c288a8cb13eb6c6819 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 82e1ecef80f449029732e3ade3c4a092, bce5b828f0a947c288a8cb13eb6c6819; no voters: d4b1269455e04063b0eb4e2aa46d0684
I20260929 03:18:03.617357 28614 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bce5b828f0a947c288a8cb13eb6c6819 [term 3 FOLLOWER]: Leader election won for term 3
I20260929 03:18:03.617523 28614 raft_consensus.cc:686] T 00000000000000000000000000000000 P bce5b828f0a947c288a8cb13eb6c6819 [term 3 LEADER]: Becoming Leader. State: Replica: bce5b828f0a947c288a8cb13eb6c6819, State: Running, Role: LEADER
I20260929 03:18:03.617619 28614 consensus_queue.cc:237] T 00000000000000000000000000000000 P bce5b828f0a947c288a8cb13eb6c6819 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d4b1269455e04063b0eb4e2aa46d0684" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 35553 } } peers { permanent_uuid: "82e1ecef80f449029732e3ade3c4a092" member_type: VOTER last_known_addr { host: "127.20.178.125" port: 35183 } attrs { promote: false } } peers { permanent_uuid: "bce5b828f0a947c288a8cb13eb6c6819" member_type: VOTER last_known_addr { host: "127.20.178.124" port: 41439 } attrs { promote: false } }
I20260929 03:18:03.618000 28617 sys_catalog.cc:455] T 00000000000000000000000000000000 P bce5b828f0a947c288a8cb13eb6c6819 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bce5b828f0a947c288a8cb13eb6c6819. Latest consensus state: current_term: 3 leader_uuid: "bce5b828f0a947c288a8cb13eb6c6819" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d4b1269455e04063b0eb4e2aa46d0684" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 35553 } } peers { permanent_uuid: "82e1ecef80f449029732e3ade3c4a092" member_type: VOTER last_known_addr { host: "127.20.178.125" port: 35183 } attrs { promote: false } } peers { permanent_uuid: "bce5b828f0a947c288a8cb13eb6c6819" member_type: VOTER last_known_addr { host: "127.20.178.124" port: 41439 } attrs { promote: false } } }
I20260929 03:18:03.618090 28617 sys_catalog.cc:458] T 00000000000000000000000000000000 P bce5b828f0a947c288a8cb13eb6c6819 [sys.catalog]: This master's current role is: LEADER
I20260929 03:18:03.618268 28619 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:18:03.618741 28619 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:18:03.618818 28619 catalog_manager.cc:1259] Loaded cluster ID: d4b12290c9754a30a1ed0ccfd1dc774e
I20260929 03:18:03.618834 28619 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:18:03.618969 28619 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:18:03.619016 28619 catalog_manager.cc:5879] T 00000000000000000000000000000000 P bce5b828f0a947c288a8cb13eb6c6819: Loaded TSK: 0
I20260929 03:18:03.619107 28619 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 03:18:03.679780 28080 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44086:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 03:18:03.681476 28080 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 03:18:03.682440 27907 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 82e1ecef80f449029732e3ade3c4a092 [term 3 FOLLOWER]: Refusing update from remote peer bce5b828f0a947c288a8cb13eb6c6819: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260929 03:18:03.682490 28615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:03.682357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:35553 (local address 127.0.0.1:43306)
0929 03:18:03.682424 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:03.682458 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.178.126:35553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260929 03:18:03.682848 28615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:03.682729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:35553 (local address 127.0.0.1:43308)
0929 03:18:03.682759 (+ 30us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:03.682784 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.178.126:35553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":14}
I20260929 03:18:03.682788 28617 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bce5b828f0a947c288a8cb13eb6c6819 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82e1ecef80f449029732e3ade3c4a092" member_type: VOTER last_known_addr { host: "127.20.178.125" port: 35183 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260929 03:18:03.683048 28067 consensus_peers.cc:487] T 00000000000000000000000000000000 P bce5b828f0a947c288a8cb13eb6c6819 -> Peer d4b1269455e04063b0eb4e2aa46d0684 (127.20.178.126:35553): Couldn't send request to peer d4b1269455e04063b0eb4e2aa46d0684. Status: Network error: Client connection negotiation failed: client connection to 127.20.178.126:35553: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260929 03:18:03.683578 28621 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82e1ecef80f449029732e3ade3c4a092 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bce5b828f0a947c288a8cb13eb6c6819. Latest consensus state: current_term: 3 leader_uuid: "bce5b828f0a947c288a8cb13eb6c6819" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d4b1269455e04063b0eb4e2aa46d0684" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 35553 } } peers { permanent_uuid: "82e1ecef80f449029732e3ade3c4a092" member_type: VOTER last_known_addr { host: "127.20.178.125" port: 35183 } attrs { promote: false } } peers { permanent_uuid: "bce5b828f0a947c288a8cb13eb6c6819" member_type: VOTER last_known_addr { host: "127.20.178.124" port: 41439 } attrs { promote: false } } }
I20260929 03:18:03.683687 28621 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82e1ecef80f449029732e3ade3c4a092 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 03:18:03.684140 28622 mvcc.cc:205] Tried to move back new op lower bound from 7334510115561521152 to 7334510115298320384. Current Snapshot: MvccSnapshot[applied={T|T < 7334510115561521152}]
I20260929 03:18:03.684131 28614 sys_catalog.cc:455] T 00000000000000000000000000000000 P bce5b828f0a947c288a8cb13eb6c6819 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "bce5b828f0a947c288a8cb13eb6c6819" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d4b1269455e04063b0eb4e2aa46d0684" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 35553 } } peers { permanent_uuid: "82e1ecef80f449029732e3ade3c4a092" member_type: VOTER last_known_addr { host: "127.20.178.125" port: 35183 } attrs { promote: false } } peers { permanent_uuid: "bce5b828f0a947c288a8cb13eb6c6819" member_type: VOTER last_known_addr { host: "127.20.178.124" port: 41439 } attrs { promote: false } } }
I20260929 03:18:03.684240 28614 sys_catalog.cc:458] T 00000000000000000000000000000000 P bce5b828f0a947c288a8cb13eb6c6819 [sys.catalog]: This master's current role is: LEADER
I20260929 03:18:03.684306 28621 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82e1ecef80f449029732e3ade3c4a092 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "bce5b828f0a947c288a8cb13eb6c6819" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d4b1269455e04063b0eb4e2aa46d0684" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 35553 } } peers { permanent_uuid: "82e1ecef80f449029732e3ade3c4a092" member_type: VOTER last_known_addr { host: "127.20.178.125" port: 35183 } attrs { promote: false } } peers { permanent_uuid: "bce5b828f0a947c288a8cb13eb6c6819" member_type: VOTER last_known_addr { host: "127.20.178.124" port: 41439 } attrs { promote: false } } }
I20260929 03:18:03.684412 28621 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82e1ecef80f449029732e3ade3c4a092 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 03:18:03.689424 28615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:03.685949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.65:45139 (local address 127.0.0.1:55482)
0929 03:18:03.685980 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:03.685991 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 03:18:03.686039 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:18:03.686447 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:18:03.686448 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:18:03.686449 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:18:03.686499 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:03.686500 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:03.687439 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:03.687440 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:03.688309 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:03.688311 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:03.689145 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:03.689146 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:03.689211 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:03.689215 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 03:18:03.689265 (+ 50us) client_negotiation.cc:240] Negotiation successful
0929 03:18:03.689301 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16,"mutex_wait_us":52}
I20260929 03:18:03.689455 28630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:03.686222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58428 (local address 127.20.178.66:42401)
0929 03:18:03.686315 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0929 03:18:03.686317 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:18:03.686325 (+ 8us) server_negotiation.cc:407] Connection header received
0929 03:18:03.686348 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:18:03.686349 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:18:03.686359 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:18:03.686457 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:18:03.686594 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:03.687265 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:03.688654 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:03.689143 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:03.689215 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:03.689225 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0929 03:18:03.689346 (+ 121us) server_negotiation.cc:299] Negotiation successful
0929 03:18:03.689387 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75,"thread_start_us":46,"threads_started":1}
I20260929 03:18:03.689630 28627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:03.686045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.66:42401 (local address 127.0.0.1:58428)
0929 03:18:03.686184 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:03.686194 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 03:18:03.686231 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:18:03.686459 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:18:03.686460 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:18:03.686462 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:18:03.686509 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:03.686511 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:03.687363 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:03.687364 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:03.688539 (+ 1175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:03.688542 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:03.689215 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:03.689216 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:03.689275 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:03.689279 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 03:18:03.689334 (+ 55us) client_negotiation.cc:240] Negotiation successful
0929 03:18:03.689373 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":86,"threads_started":1}
I20260929 03:18:03.689719 28626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:03.686063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.67:32833 (local address 127.0.0.1:37020)
0929 03:18:03.686519 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:03.686529 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 03:18:03.686576 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:18:03.686781 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:18:03.686782 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:18:03.686784 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:18:03.686817 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:03.686818 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:03.687672 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:03.687673 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:03.688741 (+ 1068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:03.688743 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:03.689506 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:03.689507 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:03.689570 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:03.689573 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 03:18:03.689619 (+ 46us) client_negotiation.cc:240] Negotiation successful
0929 03:18:03.689657 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":84,"threads_started":1}
I20260929 03:18:03.689839 28628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:03.686119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37020 (local address 127.20.178.67:32833)
0929 03:18:03.686307 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0929 03:18:03.686309 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:18:03.686630 (+ 321us) server_negotiation.cc:407] Connection header received
0929 03:18:03.686667 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:18:03.686669 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:18:03.686680 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:18:03.686740 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:18:03.686894 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:03.687593 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:03.688818 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:03.689402 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:03.689464 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:03.689474 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0929 03:18:03.689640 (+ 166us) server_negotiation.cc:299] Negotiation successful
0929 03:18:03.689771 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":49,"threads_started":1}
I20260929 03:18:03.689918 28332 tablet_service.cc:1467] Processing CreateTablet for tablet 782d2282772a4b5c96ec02148d300fea (DEFAULT_TABLE table=test-table [id=d0b311cabf3d4b0799510579df71a760]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:18:03.690029 28332 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 782d2282772a4b5c96ec02148d300fea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:18:03.690055 28468 tablet_service.cc:1467] Processing CreateTablet for tablet 782d2282772a4b5c96ec02148d300fea (DEFAULT_TABLE table=test-table [id=d0b311cabf3d4b0799510579df71a760]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:18:03.690156 28468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 782d2282772a4b5c96ec02148d300fea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:18:03.691114 28631 tablet_bootstrap.cc:492] T 782d2282772a4b5c96ec02148d300fea P 5064f9a44a034c0dadb4054ac8824780: Bootstrap starting.
I20260929 03:18:03.691390 28632 tablet_bootstrap.cc:492] T 782d2282772a4b5c96ec02148d300fea P b2e7ad7ddf214b3596c571f5cd4d3272: Bootstrap starting.
I20260929 03:18:03.691506 28631 tablet_bootstrap.cc:654] T 782d2282772a4b5c96ec02148d300fea P 5064f9a44a034c0dadb4054ac8824780: Neither blocks nor log segments found. Creating new log.
I20260929 03:18:03.691576 28629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:03.686190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55482 (local address 127.20.178.65:45139)
0929 03:18:03.686314 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0929 03:18:03.686316 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:18:03.686324 (+ 8us) server_negotiation.cc:407] Connection header received
0929 03:18:03.686348 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:18:03.686349 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:18:03.686360 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:18:03.686433 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:18:03.686564 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:03.687347 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:03.688457 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:03.689059 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:03.689130 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:03.689141 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0929 03:18:03.689407 (+ 266us) server_negotiation.cc:299] Negotiation successful
0929 03:18:03.689451 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":60,"threads_started":1}
I20260929 03:18:03.691727 28632 tablet_bootstrap.cc:654] T 782d2282772a4b5c96ec02148d300fea P b2e7ad7ddf214b3596c571f5cd4d3272: Neither blocks nor log segments found. Creating new log.
I20260929 03:18:03.691764 28195 tablet_service.cc:1467] Processing CreateTablet for tablet 782d2282772a4b5c96ec02148d300fea (DEFAULT_TABLE table=test-table [id=d0b311cabf3d4b0799510579df71a760]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:18:03.691874 28195 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 782d2282772a4b5c96ec02148d300fea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:18:03.692257 28632 tablet_bootstrap.cc:492] T 782d2282772a4b5c96ec02148d300fea P b2e7ad7ddf214b3596c571f5cd4d3272: No bootstrap required, opened a new log
I20260929 03:18:03.692319 28632 ts_tablet_manager.cc:1397] T 782d2282772a4b5c96ec02148d300fea P b2e7ad7ddf214b3596c571f5cd4d3272: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260929 03:18:03.692425 28632 raft_consensus.cc:348] T 782d2282772a4b5c96ec02148d300fea P b2e7ad7ddf214b3596c571f5cd4d3272 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e76cae1a85be4646b41cc2ed226fe5c7" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 45139 } } peers { permanent_uuid: "b2e7ad7ddf214b3596c571f5cd4d3272" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 32833 } } peers { permanent_uuid: "5064f9a44a034c0dadb4054ac8824780" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 42401 } }
I20260929 03:18:03.692504 28632 raft_consensus.cc:374] T 782d2282772a4b5c96ec02148d300fea P b2e7ad7ddf214b3596c571f5cd4d3272 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:18:03.692550 28632 raft_consensus.cc:729] T 782d2282772a4b5c96ec02148d300fea P b2e7ad7ddf214b3596c571f5cd4d3272 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2e7ad7ddf214b3596c571f5cd4d3272, State: Initialized, Role: FOLLOWER
I20260929 03:18:03.692602 28632 consensus_queue.cc:260] T 782d2282772a4b5c96ec02148d300fea P b2e7ad7ddf214b3596c571f5cd4d3272 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e76cae1a85be4646b41cc2ed226fe5c7" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 45139 } } peers { permanent_uuid: "b2e7ad7ddf214b3596c571f5cd4d3272" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 32833 } } peers { permanent_uuid: "5064f9a44a034c0dadb4054ac8824780" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 42401 } }
I20260929 03:18:03.692832 28632 ts_tablet_manager.cc:1428] T 782d2282772a4b5c96ec02148d300fea P b2e7ad7ddf214b3596c571f5cd4d3272: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260929 03:18:03.693305 28612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:03.692973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:35553 (local address 127.20.178.67:57013)
0929 03:18:03.693243 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:03.693275 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.178.126:35553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":252}
I20260929 03:18:03.693362 28534 heartbeater.cc:503] Master 127.20.178.124:41439 was elected leader, sending a full tablet report...
I20260929 03:18:03.693466 28636 tablet_bootstrap.cc:492] T 782d2282772a4b5c96ec02148d300fea P e76cae1a85be4646b41cc2ed226fe5c7: Bootstrap starting.
I20260929 03:18:03.693773 28612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:03.693636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:35553 (local address 127.20.178.67:51569)
0929 03:18:03.693723 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:03.693748 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.178.126:35553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260929 03:18:03.693817 28631 tablet_bootstrap.cc:492] T 782d2282772a4b5c96ec02148d300fea P 5064f9a44a034c0dadb4054ac8824780: No bootstrap required, opened a new log
I20260929 03:18:03.693868 28631 ts_tablet_manager.cc:1397] T 782d2282772a4b5c96ec02148d300fea P 5064f9a44a034c0dadb4054ac8824780: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260929 03:18:03.693969 28631 raft_consensus.cc:348] T 782d2282772a4b5c96ec02148d300fea P 5064f9a44a034c0dadb4054ac8824780 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e76cae1a85be4646b41cc2ed226fe5c7" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 45139 } } peers { permanent_uuid: "b2e7ad7ddf214b3596c571f5cd4d3272" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 32833 } } peers { permanent_uuid: "5064f9a44a034c0dadb4054ac8824780" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 42401 } }
I20260929 03:18:03.694029 28636 tablet_bootstrap.cc:654] T 782d2282772a4b5c96ec02148d300fea P e76cae1a85be4646b41cc2ed226fe5c7: Neither blocks nor log segments found. Creating new log.
I20260929 03:18:03.694049 28631 raft_consensus.cc:374] T 782d2282772a4b5c96ec02148d300fea P 5064f9a44a034c0dadb4054ac8824780 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:18:03.694072 28631 raft_consensus.cc:729] T 782d2282772a4b5c96ec02148d300fea P 5064f9a44a034c0dadb4054ac8824780 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5064f9a44a034c0dadb4054ac8824780, State: Initialized, Role: FOLLOWER
I20260929 03:18:03.694123 28631 consensus_queue.cc:260] T 782d2282772a4b5c96ec02148d300fea P 5064f9a44a034c0dadb4054ac8824780 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e76cae1a85be4646b41cc2ed226fe5c7" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 45139 } } peers { permanent_uuid: "b2e7ad7ddf214b3596c571f5cd4d3272" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 32833 } } peers { permanent_uuid: "5064f9a44a034c0dadb4054ac8824780" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 42401 } }
I20260929 03:18:03.694298 28631 ts_tablet_manager.cc:1428] T 782d2282772a4b5c96ec02148d300fea P 5064f9a44a034c0dadb4054ac8824780: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260929 03:18:03.694635 28611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:03.694517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:35553 (local address 127.20.178.66:51071)
0929 03:18:03.694555 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:03.694586 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.178.126:35553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
I20260929 03:18:03.694785 28398 heartbeater.cc:503] Master 127.20.178.124:41439 was elected leader, sending a full tablet report...
I20260929 03:18:03.695031 28636 tablet_bootstrap.cc:492] T 782d2282772a4b5c96ec02148d300fea P e76cae1a85be4646b41cc2ed226fe5c7: No bootstrap required, opened a new log
I20260929 03:18:03.695082 28636 ts_tablet_manager.cc:1397] T 782d2282772a4b5c96ec02148d300fea P e76cae1a85be4646b41cc2ed226fe5c7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 03:18:03.695194 28636 raft_consensus.cc:348] T 782d2282772a4b5c96ec02148d300fea P e76cae1a85be4646b41cc2ed226fe5c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e76cae1a85be4646b41cc2ed226fe5c7" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 45139 } } peers { permanent_uuid: "b2e7ad7ddf214b3596c571f5cd4d3272" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 32833 } } peers { permanent_uuid: "5064f9a44a034c0dadb4054ac8824780" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 42401 } }
I20260929 03:18:03.695274 28636 raft_consensus.cc:374] T 782d2282772a4b5c96ec02148d300fea P e76cae1a85be4646b41cc2ed226fe5c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:18:03.695310 28636 raft_consensus.cc:729] T 782d2282772a4b5c96ec02148d300fea P e76cae1a85be4646b41cc2ed226fe5c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e76cae1a85be4646b41cc2ed226fe5c7, State: Initialized, Role: FOLLOWER
I20260929 03:18:03.695366 28636 consensus_queue.cc:260] T 782d2282772a4b5c96ec02148d300fea P e76cae1a85be4646b41cc2ed226fe5c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e76cae1a85be4646b41cc2ed226fe5c7" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 45139 } } peers { permanent_uuid: "b2e7ad7ddf214b3596c571f5cd4d3272" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 32833 } } peers { permanent_uuid: "5064f9a44a034c0dadb4054ac8824780" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 42401 } }
I20260929 03:18:03.695514 28636 ts_tablet_manager.cc:1428] T 782d2282772a4b5c96ec02148d300fea P e76cae1a85be4646b41cc2ed226fe5c7: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260929 03:18:03.695780 28613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:03.695690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:35553 (local address 127.20.178.65:38743)
0929 03:18:03.695730 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:03.695755 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.178.126:35553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260929 03:18:03.696034 28261 heartbeater.cc:503] Master 127.20.178.124:41439 was elected leader, sending a full tablet report...
I20260929 03:18:03.696146 28613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:03.696047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:35553 (local address 127.20.178.65:45493)
0929 03:18:03.696077 (+ 30us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:03.696102 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.178.126:35553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
I20260929 03:18:03.790987 28638 raft_consensus.cc:482] T 782d2282772a4b5c96ec02148d300fea P 5064f9a44a034c0dadb4054ac8824780 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:18:03.791090 28638 raft_consensus.cc:504] T 782d2282772a4b5c96ec02148d300fea P 5064f9a44a034c0dadb4054ac8824780 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e76cae1a85be4646b41cc2ed226fe5c7" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 45139 } } peers { permanent_uuid: "b2e7ad7ddf214b3596c571f5cd4d3272" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 32833 } } peers { permanent_uuid: "5064f9a44a034c0dadb4054ac8824780" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 42401 } }
I20260929 03:18:03.791487 28638 leader_election.cc:288] T 782d2282772a4b5c96ec02148d300fea P 5064f9a44a034c0dadb4054ac8824780 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e76cae1a85be4646b41cc2ed226fe5c7 (127.20.178.65:45139), b2e7ad7ddf214b3596c571f5cd4d3272 (127.20.178.67:32833)
I20260929 03:18:03.794900 28628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:03.791889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.66:56937 (local address 127.20.178.67:32833)
0929 03:18:03.791953 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0929 03:18:03.791955 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:18:03.792024 (+ 69us) server_negotiation.cc:407] Connection header received
0929 03:18:03.792085 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:18:03.792086 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:18:03.792096 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:18:03.792152 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:18:03.792312 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:03.793100 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:03.794235 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:03.794589 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:03.794671 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:03.794681 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0929 03:18:03.794812 (+ 131us) server_negotiation.cc:299] Negotiation successful
0929 03:18:03.794849 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260929 03:18:03.794926 28640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:03.791790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.67:32833 (local address 127.20.178.66:56937)
0929 03:18:03.791942 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:03.791953 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 03:18:03.792023 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:18:03.792160 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:18:03.792161 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:18:03.792163 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:18:03.792221 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:03.792223 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:03.793199 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:03.793200 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:03.794127 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:03.794129 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:03.794669 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:03.794670 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:03.794726 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:03.794730 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 03:18:03.794787 (+ 57us) client_negotiation.cc:240] Negotiation successful
0929 03:18:03.794823 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"mutex_wait_us":34,"thread_start_us":105,"threads_started":1}
I20260929 03:18:03.795249 28488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "782d2282772a4b5c96ec02148d300fea" candidate_uuid: "5064f9a44a034c0dadb4054ac8824780" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2e7ad7ddf214b3596c571f5cd4d3272" is_pre_election: true
I20260929 03:18:03.795351 28488 raft_consensus.cc:2454] T 782d2282772a4b5c96ec02148d300fea P b2e7ad7ddf214b3596c571f5cd4d3272 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5064f9a44a034c0dadb4054ac8824780 in term 0.
I20260929 03:18:03.795543 28288 leader_election.cc:302] T 782d2282772a4b5c96ec02148d300fea P 5064f9a44a034c0dadb4054ac8824780 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5064f9a44a034c0dadb4054ac8824780, b2e7ad7ddf214b3596c571f5cd4d3272; no voters:
I20260929 03:18:03.795635 28611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:03.791683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.65:45139 (local address 127.20.178.66:36571)
0929 03:18:03.791928 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:03.791938 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 03:18:03.792043 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:18:03.792246 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:18:03.792247 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:18:03.792248 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:18:03.792284 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:03.792285 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:03.793187 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:03.793188 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:03.794124 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:03.794127 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:03.795415 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:03.795417 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:03.795481 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:03.795484 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 03:18:03.795538 (+ 54us) client_negotiation.cc:240] Negotiation successful
0929 03:18:03.795578 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226,"mutex_wait_us":78}
I20260929 03:18:03.795675 28629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:03.791755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.66:36571 (local address 127.20.178.65:45139)
0929 03:18:03.791820 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0929 03:18:03.791823 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:18:03.792044 (+ 221us) server_negotiation.cc:407] Connection header received
0929 03:18:03.792129 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:18:03.792130 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:18:03.792140 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:18:03.792243 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:18:03.792359 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:03.793099 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:03.794713 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:03.795290 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:03.795396 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:03.795407 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0929 03:18:03.795545 (+ 138us) server_negotiation.cc:299] Negotiation successful
0929 03:18:03.795594 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260929 03:18:03.795745 28638 raft_consensus.cc:2781] T 782d2282772a4b5c96ec02148d300fea P 5064f9a44a034c0dadb4054ac8824780 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 03:18:03.795789 28638 raft_consensus.cc:482] T 782d2282772a4b5c96ec02148d300fea P 5064f9a44a034c0dadb4054ac8824780 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 03:18:03.795820 28638 raft_consensus.cc:3037] T 782d2282772a4b5c96ec02148d300fea P 5064f9a44a034c0dadb4054ac8824780 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:18:03.795853 28215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "782d2282772a4b5c96ec02148d300fea" candidate_uuid: "5064f9a44a034c0dadb4054ac8824780" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e76cae1a85be4646b41cc2ed226fe5c7" is_pre_election: true
I20260929 03:18:03.795974 28215 raft_consensus.cc:2454] T 782d2282772a4b5c96ec02148d300fea P e76cae1a85be4646b41cc2ed226fe5c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5064f9a44a034c0dadb4054ac8824780 in term 0.
I20260929 03:18:03.796777 28638 raft_consensus.cc:504] T 782d2282772a4b5c96ec02148d300fea P 5064f9a44a034c0dadb4054ac8824780 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e76cae1a85be4646b41cc2ed226fe5c7" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 45139 } } peers { permanent_uuid: "b2e7ad7ddf214b3596c571f5cd4d3272" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 32833 } } peers { permanent_uuid: "5064f9a44a034c0dadb4054ac8824780" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 42401 } }
I20260929 03:18:03.796900 28638 leader_election.cc:288] T 782d2282772a4b5c96ec02148d300fea P 5064f9a44a034c0dadb4054ac8824780 [CANDIDATE]: Term 1 election: Requested vote from peers e76cae1a85be4646b41cc2ed226fe5c7 (127.20.178.65:45139), b2e7ad7ddf214b3596c571f5cd4d3272 (127.20.178.67:32833)
I20260929 03:18:03.797061 28488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "782d2282772a4b5c96ec02148d300fea" candidate_uuid: "5064f9a44a034c0dadb4054ac8824780" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2e7ad7ddf214b3596c571f5cd4d3272"
I20260929 03:18:03.797055 28215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "782d2282772a4b5c96ec02148d300fea" candidate_uuid: "5064f9a44a034c0dadb4054ac8824780" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e76cae1a85be4646b41cc2ed226fe5c7"
I20260929 03:18:03.797133 28215 raft_consensus.cc:3037] T 782d2282772a4b5c96ec02148d300fea P e76cae1a85be4646b41cc2ed226fe5c7 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:18:03.797133 28488 raft_consensus.cc:3037] T 782d2282772a4b5c96ec02148d300fea P b2e7ad7ddf214b3596c571f5cd4d3272 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:18:03.797894 28488 raft_consensus.cc:2454] T 782d2282772a4b5c96ec02148d300fea P b2e7ad7ddf214b3596c571f5cd4d3272 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5064f9a44a034c0dadb4054ac8824780 in term 1.
I20260929 03:18:03.797945 28215 raft_consensus.cc:2454] T 782d2282772a4b5c96ec02148d300fea P e76cae1a85be4646b41cc2ed226fe5c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5064f9a44a034c0dadb4054ac8824780 in term 1.
I20260929 03:18:03.798063 28288 leader_election.cc:302] T 782d2282772a4b5c96ec02148d300fea P 5064f9a44a034c0dadb4054ac8824780 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5064f9a44a034c0dadb4054ac8824780, e76cae1a85be4646b41cc2ed226fe5c7; no voters:
I20260929 03:18:03.798152 28638 raft_consensus.cc:2781] T 782d2282772a4b5c96ec02148d300fea P 5064f9a44a034c0dadb4054ac8824780 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:18:03.798310 28638 raft_consensus.cc:686] T 782d2282772a4b5c96ec02148d300fea P 5064f9a44a034c0dadb4054ac8824780 [term 1 LEADER]: Becoming Leader. State: Replica: 5064f9a44a034c0dadb4054ac8824780, State: Running, Role: LEADER
I20260929 03:18:03.798393 28638 consensus_queue.cc:237] T 782d2282772a4b5c96ec02148d300fea P 5064f9a44a034c0dadb4054ac8824780 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e76cae1a85be4646b41cc2ed226fe5c7" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 45139 } } peers { permanent_uuid: "b2e7ad7ddf214b3596c571f5cd4d3272" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 32833 } } peers { permanent_uuid: "5064f9a44a034c0dadb4054ac8824780" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 42401 } }
I20260929 03:18:03.798628 28611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:03.798533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:35553 (local address 127.20.178.66:39553)
0929 03:18:03.798581 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:03.798610 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.178.126:35553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260929 03:18:03.799077 28080 catalog_manager.cc:5495] T 782d2282772a4b5c96ec02148d300fea P 5064f9a44a034c0dadb4054ac8824780 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5064f9a44a034c0dadb4054ac8824780 (127.20.178.66). New cstate: current_term: 1 leader_uuid: "5064f9a44a034c0dadb4054ac8824780" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e76cae1a85be4646b41cc2ed226fe5c7" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 45139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b2e7ad7ddf214b3596c571f5cd4d3272" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 32833 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5064f9a44a034c0dadb4054ac8824780" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 42401 } health_report { overall_health: HEALTHY } } }
I20260929 03:18:03.833482 28630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:03.831016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58438 (local address 127.20.178.66:42401)
0929 03:18:03.831079 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0929 03:18:03.831081 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:18:03.831089 (+ 8us) server_negotiation.cc:407] Connection header received
0929 03:18:03.831144 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:18:03.831145 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:18:03.831162 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:18:03.831222 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:18:03.831428 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:03.831883 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:03.832810 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:03.832991 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:03.833041 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:03.833174 (+ 133us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:18:03.833270 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:18:03.833324 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0929 03:18:03.833370 (+ 46us) server_negotiation.cc:299] Negotiation successful
0929 03:18:03.833386 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260929 03:18:03.834668 28488 raft_consensus.cc:1261] T 782d2282772a4b5c96ec02148d300fea P b2e7ad7ddf214b3596c571f5cd4d3272 [term 1 FOLLOWER]: Refusing update from remote peer 5064f9a44a034c0dadb4054ac8824780: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:18:03.834703 28215 raft_consensus.cc:1261] T 782d2282772a4b5c96ec02148d300fea P e76cae1a85be4646b41cc2ed226fe5c7 [term 1 FOLLOWER]: Refusing update from remote peer 5064f9a44a034c0dadb4054ac8824780: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:18:03.834946 28641 consensus_queue.cc:1035] T 782d2282772a4b5c96ec02148d300fea P 5064f9a44a034c0dadb4054ac8824780 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2e7ad7ddf214b3596c571f5cd4d3272" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 32833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:18:03.835106 28638 consensus_queue.cc:1035] T 782d2282772a4b5c96ec02148d300fea P 5064f9a44a034c0dadb4054ac8824780 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e76cae1a85be4646b41cc2ed226fe5c7" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 45139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:18:03.836004 28612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:03.835789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:35553 (local address 127.20.178.67:45951)
0929 03:18:03.835927 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:03.835966 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.178.126:35553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119}
I20260929 03:18:03.836035 28613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:03.835871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:35553 (local address 127.20.178.65:33681)
0929 03:18:03.835965 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:03.836000 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.178.126:35553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260929 03:18:03.837217 28611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:03.837113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:35553 (local address 127.20.178.66:45909)
0929 03:18:03.837146 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:03.837188 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.178.126:35553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
I20260929 03:18:03.837248 28612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:03.837153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:35553 (local address 127.20.178.67:53977)
0929 03:18:03.837190 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:03.837215 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.178.126:35553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260929 03:18:03.837595 28611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:03.837516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:35553 (local address 127.20.178.66:44935)
0929 03:18:03.837550 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:03.837572 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.178.126:35553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
I20260929 03:18:03.837697 28646 mvcc.cc:205] Tried to move back new op lower bound from 7334510116184412160 to 7334510116038664192. Current Snapshot: MvccSnapshot[applied={T|T < 7334510116184412160 or (T in {7334510116184412160})}]
I20260929 03:18:03.838356 28613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:03.838182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:35553 (local address 127.20.178.65:44513)
0929 03:18:03.838303 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:03.838329 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.178.126:35553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
I20260929 03:18:03.839766 28081 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44086:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:18:03.839851 28081 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44086:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:18:03.841063 28081 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bce5b828f0a947c288a8cb13eb6c6819: Sending DeleteTablet for 3 replicas of tablet 782d2282772a4b5c96ec02148d300fea
I20260929 03:18:03.841375 28195 tablet_service.cc:1514] Processing DeleteTablet for tablet 782d2282772a4b5c96ec02148d300fea with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:18:03 UTC) from {username='oryx', principal='oryx/127.20.178.124@KRBTEST.COM'} at 127.0.0.1:55482
I20260929 03:18:03.841377 28468 tablet_service.cc:1514] Processing DeleteTablet for tablet 782d2282772a4b5c96ec02148d300fea with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:18:03 UTC) from {username='oryx', principal='oryx/127.20.178.124@KRBTEST.COM'} at 127.0.0.1:37020
I20260929 03:18:03.841492 28332 tablet_service.cc:1514] Processing DeleteTablet for tablet 782d2282772a4b5c96ec02148d300fea with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:18:03 UTC) from {username='oryx', principal='oryx/127.20.178.124@KRBTEST.COM'} at 127.0.0.1:58428
I20260929 03:18:03.841677 28656 tablet_replica.cc:330] T 782d2282772a4b5c96ec02148d300fea P 5064f9a44a034c0dadb4054ac8824780: stopping tablet replica
I20260929 03:18:03.841676 28654 tablet_replica.cc:330] T 782d2282772a4b5c96ec02148d300fea P e76cae1a85be4646b41cc2ed226fe5c7: stopping tablet replica
I20260929 03:18:03.841737 28656 raft_consensus.cc:2229] T 782d2282772a4b5c96ec02148d300fea P 5064f9a44a034c0dadb4054ac8824780 [term 1 LEADER]: Raft consensus shutting down.
I20260929 03:18:03.841832 28656 raft_consensus.cc:2258] T 782d2282772a4b5c96ec02148d300fea P 5064f9a44a034c0dadb4054ac8824780 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:18:03.841833 21193 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuoaHbX/build/release/bin/kudu with pid 28044
I20260929 03:18:03.844997 28656 ts_tablet_manager.cc:1905] T 782d2282772a4b5c96ec02148d300fea P 5064f9a44a034c0dadb4054ac8824780: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:18:03.845404 28655 tablet_replica.cc:330] T 782d2282772a4b5c96ec02148d300fea P b2e7ad7ddf214b3596c571f5cd4d3272: stopping tablet replica
I20260929 03:18:03.845474 28655 raft_consensus.cc:2229] T 782d2282772a4b5c96ec02148d300fea P b2e7ad7ddf214b3596c571f5cd4d3272 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:18:03.845525 28655 raft_consensus.cc:2258] T 782d2282772a4b5c96ec02148d300fea P b2e7ad7ddf214b3596c571f5cd4d3272 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:18:03.847088 28656 ts_tablet_manager.cc:1918] T 782d2282772a4b5c96ec02148d300fea P 5064f9a44a034c0dadb4054ac8824780: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:18:03.847152 28656 log.cc:1196] T 782d2282772a4b5c96ec02148d300fea P 5064f9a44a034c0dadb4054ac8824780: Deleting WAL directory at /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/minicluster-data/ts-1/wal/wals/782d2282772a4b5c96ec02148d300fea
I20260929 03:18:03.847391 28656 ts_tablet_manager.cc:1939] T 782d2282772a4b5c96ec02148d300fea P 5064f9a44a034c0dadb4054ac8824780: Deleting consensus metadata
I20260929 03:18:03.847705 28066 catalog_manager.cc:4841] TS 5064f9a44a034c0dadb4054ac8824780 (127.20.178.66:42401): tablet 782d2282772a4b5c96ec02148d300fea (table test-table [id=d0b311cabf3d4b0799510579df71a760]) successfully deleted
W20260929 03:18:03.848615 28626 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:18:03.847979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.65:45139 (local address 127.0.0.1:55490)
0929 03:18:03.848039 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:03.848052 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 03:18:03.848093 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:18:03.848565 (+ 472us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.178.65:45139: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":40}
W20260929 03:18:03.848801 28069 catalog_manager.cc:4584] TS e76cae1a85be4646b41cc2ed226fe5c7 (127.20.178.65:45139): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 782d2282772a4b5c96ec02148d300fea: Network error: Client connection negotiation failed: client connection to 127.20.178.65:45139: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260929 03:18:03.849074 21193 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuoaHbX/build/release/bin/kudu with pid 28269
I20260929 03:18:03.849705 28655 ts_tablet_manager.cc:1905] T 782d2282772a4b5c96ec02148d300fea P b2e7ad7ddf214b3596c571f5cd4d3272: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:18:03.850912 28655 ts_tablet_manager.cc:1918] T 782d2282772a4b5c96ec02148d300fea P b2e7ad7ddf214b3596c571f5cd4d3272: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:18:03.850989 28655 log.cc:1196] T 782d2282772a4b5c96ec02148d300fea P b2e7ad7ddf214b3596c571f5cd4d3272: Deleting WAL directory at /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651852476208-21193-0/minicluster-data/ts-2/wal/wals/782d2282772a4b5c96ec02148d300fea
I20260929 03:18:03.851204 28655 ts_tablet_manager.cc:1939] T 782d2282772a4b5c96ec02148d300fea P b2e7ad7ddf214b3596c571f5cd4d3272: Deleting consensus metadata
I20260929 03:18:03.851521 28069 catalog_manager.cc:4841] TS b2e7ad7ddf214b3596c571f5cd4d3272 (127.20.178.67:32833): tablet 782d2282772a4b5c96ec02148d300fea (table test-table [id=d0b311cabf3d4b0799510579df71a760]) successfully deleted
I20260929 03:18:03.855639 21193 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuoaHbX/build/release/bin/kudu with pid 28405
I20260929 03:18:03.860255 21193 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuoaHbX/build/release/bin/kudu with pid 27762
I20260929 03:18:03.864950 21193 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuoaHbX/build/release/bin/kudu with pid 27951
2026-09-29T03:18:03Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (6684 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651852476208-21193-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:18:03 dist-test-slave-c9qh krb5kdc[28659](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:18:03 dist-test-slave-c9qh krb5kdc[28659](info): set up 2 sockets
Sep 29 03:18:03 dist-test-slave-c9qh krb5kdc[28659](info): commencing operation
krb5kdc: starting...
W20260929 03:18:04.949061 21193 mini_kdc.cc:121] Time spent starting KDC: real 1.062s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651852476208-21193-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651852476208-21193-0/krb5kdc/test-user.keytab.
Sep 29 03:18:05 dist-test-slave-c9qh krb5kdc[28659](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651885, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:18:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:18:05Z Disabled control of system clock
WARNING: no policy specified for oryx/127.20.178.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.20.178.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.20.178.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651852476208-21193-0/krb5kdc/oryx_127.20.178.126.keytab.
Entry for principal oryx/127.20.178.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651852476208-21193-0/krb5kdc/oryx_127.20.178.126.keytab.
I20260929 03:18:05.064137 21193 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuoaHbX/build/release/bin/kudu
/tmp/dist-test-taskuoaHbX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651852476208-21193-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651852476208-21193-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651852476208-21193-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651852476208-21193-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.178.126:42107
--webserver_interface=127.20.178.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651852476208-21193-0/krb5kdc/oryx_127.20.178.126.keytab
--principal=oryx/127.20.178.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.178.84:46725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.178.126:42107
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651852476208-21193-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651852476208-21193-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651852476208-21193-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:18:05.143098 28672 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:18:05.143339 28672 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:18:05.143378 28672 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:18:05.144526 28672 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:18:05.144596 28672 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260929 03:18:05.144619 28672 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:18:05.144645 28672 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:18:05.144716 28672 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:18:05.144747 28672 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:18:05.146319 28672 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.178.84:46725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651852476208-21193-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651852476208-21193-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.178.126:42107
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651852476208-21193-0/krb5kdc/oryx_127.20.178.126.keytab
--principal=oryx/127.20.178.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.178.126:42107
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651852476208-21193-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.178.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28672
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651852476208-21193-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:18:05.146595 28672 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:18:05.146817 28672 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:18:05.151556 28677 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:18:05.151523 28678 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:18:05.151588 28672 server_base.cc:1034] running on GCE node
W20260929 03:18:05.151527 28680 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:18:05.151980 28672 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:18:05.152272 28672 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:18:05.153462 28672 hybrid_clock.cc:648] HybridClock initialized: now 1790651885153459 us; error 39 us; skew 500 ppm
Sep 29 03:18:05 dist-test-slave-c9qh krb5kdc[28659](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651885, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.178.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:18:05.155786 28672 init.cc:377] Logged in from keytab as oryx/127.20.178.126@KRBTEST.COM (short username oryx)
I20260929 03:18:05.156263 28672 webserver.cc:460] Webserver started at http://127.20.178.126:35859/ using document root <none> and password file <none>
I20260929 03:18:05.156495 28672 fs_manager.cc:359] Metadata directory not provided
I20260929 03:18:05.156563 28672 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:18:05.156735 28672 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:18:05.157768 28672 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651852476208-21193-0/minicluster-data/master-0/data/instance:
uuid: "b8fefe3633e6491689c51b85ebdc7336"
format_stamp: "Formatted at 2026-09-29 03:18:05 on dist-test-slave-c9qh"
server_key: "85df5bd758cdf41c62b703a8d63b1e61"
server_key_iv: "441531481b2e4e4a96991838095597e4"
server_key_version: "encryptionkey@0"
I20260929 03:18:05.158119 28672 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651852476208-21193-0/minicluster-data/master-0/wal/instance:
uuid: "b8fefe3633e6491689c51b85ebdc7336"
format_stamp: "Formatted at 2026-09-29 03:18:05 on dist-test-slave-c9qh"
server_key: "85df5bd758cdf41c62b703a8d63b1e61"
server_key_iv: "441531481b2e4e4a96991838095597e4"
server_key_version: "encryptionkey@0"
I20260929 03:18:05.159443 28672 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 03:18:05.160147 28687 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:18:05.160341 28672 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 03:18:05.160437 28672 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651852476208-21193-0/minicluster-data/master-0/data,/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651852476208-21193-0/minicluster-data/master-0/wal
uuid: "b8fefe3633e6491689c51b85ebdc7336"
format_stamp: "Formatted at 2026-09-29 03:18:05 on dist-test-slave-c9qh"
server_key: "85df5bd758cdf41c62b703a8d63b1e61"
server_key_iv: "441531481b2e4e4a96991838095597e4"
server_key_version: "encryptionkey@0"
I20260929 03:18:05.160517 28672 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651852476208-21193-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651852476208-21193-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651852476208-21193-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:18:05.171819 28672 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:18:05.173604 28672 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:18:05.173760 28672 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:18:05.178318 28672 rpc_server.cc:307] RPC server started. Bound to: 127.20.178.126:42107
I20260929 03:18:05.178357 28749 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.178.126:42107 every 8 connection(s)
I20260929 03:18:05.178679 28672 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651852476208-21193-0/minicluster-data/master-0/data/info.pb
I20260929 03:18:05.179361 28750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:18:05.181591 28750 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8fefe3633e6491689c51b85ebdc7336: Bootstrap starting.
I20260929 03:18:05.182186 28750 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b8fefe3633e6491689c51b85ebdc7336: Neither blocks nor log segments found. Creating new log.
I20260929 03:18:05.182471 28750 log.cc:824] T 00000000000000000000000000000000 P b8fefe3633e6491689c51b85ebdc7336: Log is configured to *not* fsync() on all Append() calls
I20260929 03:18:05.183115 28750 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8fefe3633e6491689c51b85ebdc7336: No bootstrap required, opened a new log
I20260929 03:18:05.185192 28750 raft_consensus.cc:348] T 00000000000000000000000000000000 P b8fefe3633e6491689c51b85ebdc7336 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8fefe3633e6491689c51b85ebdc7336" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 42107 } }
I20260929 03:18:05.185346 28750 raft_consensus.cc:374] T 00000000000000000000000000000000 P b8fefe3633e6491689c51b85ebdc7336 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:18:05.185392 28750 raft_consensus.cc:729] T 00000000000000000000000000000000 P b8fefe3633e6491689c51b85ebdc7336 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8fefe3633e6491689c51b85ebdc7336, State: Initialized, Role: FOLLOWER
I20260929 03:18:05.185531 28750 consensus_queue.cc:260] T 00000000000000000000000000000000 P b8fefe3633e6491689c51b85ebdc7336 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8fefe3633e6491689c51b85ebdc7336" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 42107 } }
I20260929 03:18:05.185609 28750 raft_consensus.cc:388] T 00000000000000000000000000000000 P b8fefe3633e6491689c51b85ebdc7336 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:18:05.185654 28750 raft_consensus.cc:482] T 00000000000000000000000000000000 P b8fefe3633e6491689c51b85ebdc7336 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:18:05.185708 28750 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b8fefe3633e6491689c51b85ebdc7336 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:18:05.186205 28750 raft_consensus.cc:504] T 00000000000000000000000000000000 P b8fefe3633e6491689c51b85ebdc7336 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8fefe3633e6491689c51b85ebdc7336" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 42107 } }
I20260929 03:18:05.186316 28750 leader_election.cc:302] T 00000000000000000000000000000000 P b8fefe3633e6491689c51b85ebdc7336 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b8fefe3633e6491689c51b85ebdc7336; no voters:
I20260929 03:18:05.186492 28750 leader_election.cc:288] T 00000000000000000000000000000000 P b8fefe3633e6491689c51b85ebdc7336 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:18:05.186545 28753 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b8fefe3633e6491689c51b85ebdc7336 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:18:05.186748 28753 raft_consensus.cc:686] T 00000000000000000000000000000000 P b8fefe3633e6491689c51b85ebdc7336 [term 1 LEADER]: Becoming Leader. State: Replica: b8fefe3633e6491689c51b85ebdc7336, State: Running, Role: LEADER
I20260929 03:18:05.186745 28750 sys_catalog.cc:564] T 00000000000000000000000000000000 P b8fefe3633e6491689c51b85ebdc7336 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:18:05.186883 28753 consensus_queue.cc:237] T 00000000000000000000000000000000 P b8fefe3633e6491689c51b85ebdc7336 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8fefe3633e6491689c51b85ebdc7336" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 42107 } }
I20260929 03:18:05.187237 28754 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8fefe3633e6491689c51b85ebdc7336 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b8fefe3633e6491689c51b85ebdc7336" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8fefe3633e6491689c51b85ebdc7336" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 42107 } } }
I20260929 03:18:05.187247 28755 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8fefe3633e6491689c51b85ebdc7336 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b8fefe3633e6491689c51b85ebdc7336. Latest consensus state: current_term: 1 leader_uuid: "b8fefe3633e6491689c51b85ebdc7336" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8fefe3633e6491689c51b85ebdc7336" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 42107 } } }
I20260929 03:18:05.187350 28755 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8fefe3633e6491689c51b85ebdc7336 [sys.catalog]: This master's current role is: LEADER
I20260929 03:18:05.187495 28754 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8fefe3633e6491689c51b85ebdc7336 [sys.catalog]: This master's current role is: LEADER
I20260929 03:18:05.187538 28761 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:18:05.188050 28761 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:18:05.188716 21193 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuoaHbX/build/release/bin/kudu as pid 28672
I20260929 03:18:05.188814 21193 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651852476208-21193-0/minicluster-data/master-0/wal/instance
I20260929 03:18:05.189002 21193 external_mini_cluster.cc:1428] Setting key aff571fd72e7de36489d2982fc11344b
I20260929 03:18:05.189611 28761 catalog_manager.cc:1347] Generated new cluster ID: 3be010c3ff084a34b3ca2f60547e2e06
I20260929 03:18:05.189721 28761 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 29 03:18:05 dist-test-slave-c9qh krb5kdc[28659](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651885, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.20.178.126@KRBTEST.COM
I20260929 03:18:05.200348 28773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:05.189424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57088 (local address 127.20.178.126:42107)
0929 03:18:05.189788 (+ 364us) server_negotiation.cc:206] Beginning negotiation
0929 03:18:05.189797 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 03:18:05.189828 (+ 31us) server_negotiation.cc:407] Connection header received
0929 03:18:05.189937 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:18:05.189944 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:18:05.189995 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:18:05.190119 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:18:05.190873 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:05.192223 (+ 1350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:05.193367 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:05.193964 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:05.194066 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:05.196824 (+ 2758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:18:05.196828 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:18:05.196828 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:18:05.196840 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:18:05.198744 (+ 1904us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:18:05.199142 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:18:05.199145 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:18:05.199145 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:18:05.199255 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:18:05.199456 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:18:05.199457 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:18:05.199457 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:18:05.199760 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:18:05.199912 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0929 03:18:05.199980 (+ 68us) server_negotiation.cc:299] Negotiation successful
0929 03:18:05.200109 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":66,"threads_started":1}
I20260929 03:18:05.203966 28761 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:18:05.204456 28761 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:18:05.208012 28761 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b8fefe3633e6491689c51b85ebdc7336: Generated new TSK 0
I20260929 03:18:05.208199 28761 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.20.178.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.20.178.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.20.178.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651852476208-21193-0/krb5kdc/oryx_127.20.178.65.keytab.
Entry for principal oryx/127.20.178.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651852476208-21193-0/krb5kdc/oryx_127.20.178.65.keytab.
I20260929 03:18:05.236001 21193 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuoaHbX/build/release/bin/kudu
/tmp/dist-test-taskuoaHbX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651852476208-21193-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651852476208-21193-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651852476208-21193-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651852476208-21193-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.178.65:0
--local_ip_for_outbound_sockets=127.20.178.65
--webserver_interface=127.20.178.65
--webserver_port=0
--tserver_master_addrs=127.20.178.126:42107
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651852476208-21193-0/krb5kdc/oryx_127.20.178.65.keytab
--principal=oryx/127.20.178.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.178.84:46725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651852476208-21193-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651852476208-21193-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651852476208-21193-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:18:05.316721 28776 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:18:05.316913 28776 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:18:05.316936 28776 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:18:05.318102 28776 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:18:05.318143 28776 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:18:05.318164 28776 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 03:18:05.318224 28776 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.178.65
I20260929 03:18:05.319749 28776 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.178.84:46725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651852476208-21193-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651852476208-21193-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651852476208-21193-0/krb5kdc/oryx_127.20.178.65.keytab
--principal=oryx/127.20.178.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.178.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651852476208-21193-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.178.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.20.178.126:42107
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28776
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.178.65
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651852476208-21193-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:18:05.320001 28776 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:18:05.320241 28776 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:18:05.325184 28781 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:18:05.325202 28776 server_base.cc:1034] running on GCE node
W20260929 03:18:05.325186 28782 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:18:05.325186 28784 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:18:05.325539 28776 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:18:05.325791 28776 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:18:05.326954 28776 hybrid_clock.cc:648] HybridClock initialized: now 1790651885326929 us; error 58 us; skew 500 ppm
Sep 29 03:18:05 dist-test-slave-c9qh krb5kdc[28659](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651885, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.178.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:18:05.329540 28776 init.cc:377] Logged in from keytab as oryx/127.20.178.65@KRBTEST.COM (short username oryx)
I20260929 03:18:05.329980 28776 webserver.cc:460] Webserver started at http://127.20.178.65:46007/ using document root <none> and password file <none>
I20260929 03:18:05.330168 28776 fs_manager.cc:359] Metadata directory not provided
I20260929 03:18:05.330209 28776 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:18:05.330331 28776 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:18:05.331374 28776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651852476208-21193-0/minicluster-data/ts-0/data/instance:
uuid: "8bb9788606a844aeac11ef74e3475047"
format_stamp: "Formatted at 2026-09-29 03:18:05 on dist-test-slave-c9qh"
server_key: "27ef9640f7a5283ace5214833a2d7cb4"
server_key_iv: "7e28c36526f590f803843e6adb0f4544"
server_key_version: "encryptionkey@0"
I20260929 03:18:05.331720 28776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651852476208-21193-0/minicluster-data/ts-0/wal/instance:
uuid: "8bb9788606a844aeac11ef74e3475047"
format_stamp: "Formatted at 2026-09-29 03:18:05 on dist-test-slave-c9qh"
server_key: "27ef9640f7a5283ace5214833a2d7cb4"
server_key_iv: "7e28c36526f590f803843e6adb0f4544"
server_key_version: "encryptionkey@0"
I20260929 03:18:05.333092 28776 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 03:18:05.333817 28791 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:18:05.333995 28776 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 03:18:05.334084 28776 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651852476208-21193-0/minicluster-data/ts-0/data,/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651852476208-21193-0/minicluster-data/ts-0/wal
uuid: "8bb9788606a844aeac11ef74e3475047"
format_stamp: "Formatted at 2026-09-29 03:18:05 on dist-test-slave-c9qh"
server_key: "27ef9640f7a5283ace5214833a2d7cb4"
server_key_iv: "7e28c36526f590f803843e6adb0f4544"
server_key_version: "encryptionkey@0"
I20260929 03:18:05.334156 28776 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651852476208-21193-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651852476208-21193-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651852476208-21193-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:18:05.342792 28776 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:18:05.345046 28776 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:18:05.345203 28776 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:18:05.345755 28776 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:18:05.345803 28776 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:18:05.345826 28776 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:18:05.345839 28776 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 03:18:05 dist-test-slave-c9qh krb5kdc[28659](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651885, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.178.65@KRBTEST.COM for oryx/127.20.178.126@KRBTEST.COM
I20260929 03:18:05.352993 28776 rpc_server.cc:307] RPC server started. Bound to: 127.20.178.65:46097
I20260929 03:18:05.353111 28906 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.178.65:46097 every 8 connection(s)
I20260929 03:18:05.353420 28776 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651852476208-21193-0/minicluster-data/ts-0/data/info.pb
I20260929 03:18:05.355535 28773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:05.346080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.65:51811 (local address 127.20.178.126:42107)
0929 03:18:05.346151 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0929 03:18:05.346154 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:18:05.346359 (+ 205us) server_negotiation.cc:407] Connection header received
0929 03:18:05.346811 (+ 452us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:18:05.346812 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:18:05.346826 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:18:05.346903 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:18:05.347467 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:05.348270 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:05.349956 (+ 1686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:05.350167 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:05.350265 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:05.352862 (+ 2597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:18:05.352864 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:18:05.352864 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:18:05.352871 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:18:05.354448 (+ 1577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:18:05.354798 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:18:05.354798 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:18:05.354799 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:18:05.354849 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:18:05.355050 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:18:05.355051 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:18:05.355051 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:18:05.355145 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:18:05.355195 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0929 03:18:05.355411 (+ 216us) server_negotiation.cc:299] Negotiation successful
0929 03:18:05.355473 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260929 03:18:05.355712 28800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:05.346073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:42107 (local address 127.20.178.65:51811)
0929 03:18:05.346232 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:05.346270 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 03:18:05.346717 (+ 447us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:18:05.346957 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:18:05.346961 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:18:05.347178 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:18:05.347367 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:05.347371 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:05.348362 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:05.348363 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:05.349858 (+ 1495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:05.349862 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:05.350287 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:05.350288 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:05.350431 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:05.350980 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:18:05.350992 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:18:05.352742 (+ 1750us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:18:05.354519 (+ 1777us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:18:05.354523 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:18:05.354524 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:18:05.354731 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:18:05.354906 (+ 175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:18:05.354907 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:18:05.354907 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:18:05.354997 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:18:05.355197 (+ 200us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:18:05.355201 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:18:05.355305 (+ 104us) client_negotiation.cc:769] Sending connection context
0929 03:18:05.355384 (+ 79us) client_negotiation.cc:240] Negotiation successful
0929 03:18:05.355494 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":73,"threads_started":1}
I20260929 03:18:05.356230 28907 heartbeater.cc:347] Connected to a master server at 127.20.178.126:42107
I20260929 03:18:05.356302 28907 heartbeater.cc:464] Registering TS with master...
I20260929 03:18:05.356473 28907 heartbeater.cc:511] Master 127.20.178.126:42107 requested a full tablet report, sending...
I20260929 03:18:05.356984 28703 ts_manager.cc:194] Registered new tserver with Master: 8bb9788606a844aeac11ef74e3475047 (127.20.178.65:46097)
I20260929 03:18:05.357841 28703 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.20.178.65@KRBTEST.COM'} at 127.20.178.65:51811
I20260929 03:18:05.360847 21193 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuoaHbX/build/release/bin/kudu as pid 28776
I20260929 03:18:05.360963 21193 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651852476208-21193-0/minicluster-data/ts-0/wal/instance
I20260929 03:18:05.361166 21193 external_mini_cluster.cc:1428] Setting key 0dc5bc6add8f0210e4783ea91007569e
I20260929 03:18:05.362749 28773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:05.357301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.65:57857 (local address 127.20.178.126:42107)
0929 03:18:05.357342 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0929 03:18:05.357344 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:18:05.357414 (+ 70us) server_negotiation.cc:407] Connection header received
0929 03:18:05.357476 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:18:05.357476 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:18:05.357488 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:18:05.357541 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:18:05.357800 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:05.358418 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:05.359128 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:05.359301 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:05.359349 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:05.360166 (+ 817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:18:05.360167 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:18:05.360167 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:18:05.360171 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:18:05.361580 (+ 1409us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:18:05.361928 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:18:05.361929 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:18:05.361929 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:18:05.361987 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:18:05.362210 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:18:05.362211 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:18:05.362211 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:18:05.362336 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:18:05.362407 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0929 03:18:05.362576 (+ 169us) server_negotiation.cc:299] Negotiation successful
0929 03:18:05.362651 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260929 03:18:05.362757 28913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:05.357245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:42107 (local address 127.20.178.65:57857)
0929 03:18:05.357355 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:05.357364 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 03:18:05.357422 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:18:05.357544 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:18:05.357544 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:18:05.357677 (+ 133us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:18:05.357738 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:05.357739 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:05.358516 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:05.358517 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:05.359061 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:05.359063 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:05.359362 (+ 299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:05.359363 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:05.359409 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:05.359804 (+ 395us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:18:05.359809 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:18:05.360099 (+ 290us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:18:05.361653 (+ 1554us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:18:05.361654 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:18:05.361655 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:18:05.361855 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:18:05.362063 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:18:05.362064 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:18:05.362064 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:18:05.362130 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:18:05.362410 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:18:05.362412 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:18:05.362455 (+ 43us) client_negotiation.cc:769] Sending connection context
0929 03:18:05.362552 (+ 97us) client_negotiation.cc:240] Negotiation successful
0929 03:18:05.362651 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":88,"thread_start_us":60,"threads_started":1}
WARNING: no policy specified for oryx/127.20.178.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.20.178.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.20.178.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651852476208-21193-0/krb5kdc/oryx_127.20.178.66.keytab.
Entry for principal oryx/127.20.178.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651852476208-21193-0/krb5kdc/oryx_127.20.178.66.keytab.
I20260929 03:18:05.385744 21193 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuoaHbX/build/release/bin/kudu
/tmp/dist-test-taskuoaHbX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651852476208-21193-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651852476208-21193-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651852476208-21193-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651852476208-21193-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.178.66:0
--local_ip_for_outbound_sockets=127.20.178.66
--webserver_interface=127.20.178.66
--webserver_port=0
--tserver_master_addrs=127.20.178.126:42107
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651852476208-21193-0/krb5kdc/oryx_127.20.178.66.keytab
--principal=oryx/127.20.178.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.178.84:46725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651852476208-21193-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651852476208-21193-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651852476208-21193-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:18:05.466110 28916 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:18:05.466327 28916 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:18:05.466352 28916 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:18:05.467442 28916 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:18:05.467475 28916 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:18:05.467495 28916 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 03:18:05.467512 28916 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.178.66
I20260929 03:18:05.468945 28916 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.178.84:46725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651852476208-21193-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651852476208-21193-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651852476208-21193-0/krb5kdc/oryx_127.20.178.66.keytab
--principal=oryx/127.20.178.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.178.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651852476208-21193-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.178.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.20.178.126:42107
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28916
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.178.66
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651852476208-21193-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:18:05.469127 28916 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:18:05.469301 28916 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:18:05.474255 28924 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:18:05.474318 28921 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:18:05.474303 28922 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:18:05.474438 28916 server_base.cc:1034] running on GCE node
I20260929 03:18:05.474679 28916 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:18:05.474913 28916 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:18:05.476063 28916 hybrid_clock.cc:648] HybridClock initialized: now 1790651885476053 us; error 31 us; skew 500 ppm
Sep 29 03:18:05 dist-test-slave-c9qh krb5kdc[28659](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651885, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.178.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:18:05.478654 28916 init.cc:377] Logged in from keytab as oryx/127.20.178.66@KRBTEST.COM (short username oryx)
I20260929 03:18:05.479028 28916 webserver.cc:460] Webserver started at http://127.20.178.66:35247/ using document root <none> and password file <none>
I20260929 03:18:05.479213 28916 fs_manager.cc:359] Metadata directory not provided
I20260929 03:18:05.479252 28916 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:18:05.479338 28916 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:18:05.480302 28916 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651852476208-21193-0/minicluster-data/ts-1/data/instance:
uuid: "c8ef706c3c2141aa895ff4a4b0b3a3da"
format_stamp: "Formatted at 2026-09-29 03:18:05 on dist-test-slave-c9qh"
server_key: "34f35c4ae5e45f187c879dfd162e338d"
server_key_iv: "6f3b1c25084d2e26a683d0565ec3190c"
server_key_version: "encryptionkey@0"
I20260929 03:18:05.480613 28916 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651852476208-21193-0/minicluster-data/ts-1/wal/instance:
uuid: "c8ef706c3c2141aa895ff4a4b0b3a3da"
format_stamp: "Formatted at 2026-09-29 03:18:05 on dist-test-slave-c9qh"
server_key: "34f35c4ae5e45f187c879dfd162e338d"
server_key_iv: "6f3b1c25084d2e26a683d0565ec3190c"
server_key_version: "encryptionkey@0"
I20260929 03:18:05.481932 28916 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 03:18:05.482596 28931 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:18:05.482774 28916 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 03:18:05.482874 28916 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651852476208-21193-0/minicluster-data/ts-1/data,/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651852476208-21193-0/minicluster-data/ts-1/wal
uuid: "c8ef706c3c2141aa895ff4a4b0b3a3da"
format_stamp: "Formatted at 2026-09-29 03:18:05 on dist-test-slave-c9qh"
server_key: "34f35c4ae5e45f187c879dfd162e338d"
server_key_iv: "6f3b1c25084d2e26a683d0565ec3190c"
server_key_version: "encryptionkey@0"
I20260929 03:18:05.482947 28916 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651852476208-21193-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651852476208-21193-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651852476208-21193-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:18:05.499261 28916 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:18:05.501173 28916 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:18:05.501336 28916 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:18:05.501843 28916 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:18:05.501909 28916 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:18:05.501953 28916 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:18:05.501983 28916 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 03:18:05 dist-test-slave-c9qh krb5kdc[28659](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651885, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.178.66@KRBTEST.COM for oryx/127.20.178.126@KRBTEST.COM
I20260929 03:18:05.509130 28916 rpc_server.cc:307] RPC server started. Bound to: 127.20.178.66:44909
I20260929 03:18:05.509214 29046 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.178.66:44909 every 8 connection(s)
I20260929 03:18:05.509541 28916 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651852476208-21193-0/minicluster-data/ts-1/data/info.pb
I20260929 03:18:05.510584 21193 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuoaHbX/build/release/bin/kudu as pid 28916
I20260929 03:18:05.510702 21193 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651852476208-21193-0/minicluster-data/ts-1/wal/instance
I20260929 03:18:05.510907 21193 external_mini_cluster.cc:1428] Setting key 1ed97660cfce753256adb7d73c0419a7
I20260929 03:18:05.512286 28773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:05.502333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.66:50671 (local address 127.20.178.126:42107)
0929 03:18:05.502373 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0929 03:18:05.502375 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:18:05.502589 (+ 214us) server_negotiation.cc:407] Connection header received
0929 03:18:05.503112 (+ 523us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:18:05.503114 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:18:05.503125 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:18:05.503321 (+ 196us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:18:05.504178 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:05.505034 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:05.506063 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:05.506380 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:05.506452 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:05.509331 (+ 2879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:18:05.509333 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:18:05.509333 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:18:05.509342 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:18:05.510813 (+ 1471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:18:05.511282 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:18:05.511283 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:18:05.511284 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:18:05.511346 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:18:05.511632 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:18:05.511633 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:18:05.511634 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:18:05.511756 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:18:05.511830 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0929 03:18:05.512106 (+ 276us) server_negotiation.cc:299] Negotiation successful
0929 03:18:05.512185 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260929 03:18:05.512477 28940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:05.502327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:42107 (local address 127.20.178.66:50671)
0929 03:18:05.502475 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:05.502508 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0929 03:18:05.503005 (+ 497us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:18:05.503323 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:18:05.503328 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:18:05.503756 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:18:05.504062 (+ 306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:05.504070 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:05.505127 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:05.505128 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:05.505964 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:05.505967 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:05.506488 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:05.506489 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:05.506682 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:05.507251 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:18:05.507260 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:18:05.509228 (+ 1968us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:18:05.510896 (+ 1668us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:18:05.510903 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:18:05.510905 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:18:05.511188 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:18:05.511409 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:18:05.511411 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:18:05.511411 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:18:05.511544 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:18:05.511830 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:18:05.511838 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:18:05.511976 (+ 138us) client_negotiation.cc:769] Sending connection context
0929 03:18:05.512084 (+ 108us) client_negotiation.cc:240] Negotiation successful
0929 03:18:05.512244 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":83,"threads_started":1}
I20260929 03:18:05.512964 29047 heartbeater.cc:347] Connected to a master server at 127.20.178.126:42107
I20260929 03:18:05.513072 29047 heartbeater.cc:464] Registering TS with master...
I20260929 03:18:05.513281 29047 heartbeater.cc:511] Master 127.20.178.126:42107 requested a full tablet report, sending...
I20260929 03:18:05.513670 28704 ts_manager.cc:194] Registered new tserver with Master: c8ef706c3c2141aa895ff4a4b0b3a3da (127.20.178.66:44909)
I20260929 03:18:05.514144 28704 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.20.178.66@KRBTEST.COM'} at 127.20.178.66:50671
WARNING: no policy specified for oryx/127.20.178.67@KRBTEST.COM; defaulting to no policy
I20260929 03:18:05.520254 28773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:05.514005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.66:54579 (local address 127.20.178.126:42107)
0929 03:18:05.514279 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0929 03:18:05.514281 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:18:05.514289 (+ 8us) server_negotiation.cc:407] Connection header received
0929 03:18:05.514308 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:18:05.514308 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:18:05.514318 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:18:05.514365 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:18:05.514732 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:05.515526 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:05.516398 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:05.516574 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:05.516708 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:05.517724 (+ 1016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:18:05.517725 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:18:05.517725 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:18:05.517731 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:18:05.519179 (+ 1448us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:18:05.519539 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:18:05.519539 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:18:05.519540 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:18:05.519593 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:18:05.519812 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:18:05.519813 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:18:05.519813 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:18:05.519938 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:18:05.519991 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0929 03:18:05.520141 (+ 150us) server_negotiation.cc:299] Negotiation successful
0929 03:18:05.520196 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":254}
I20260929 03:18:05.520256 29054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:05.513948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:42107 (local address 127.20.178.66:54579)
0929 03:18:05.514069 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:05.514081 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 03:18:05.514138 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:18:05.514385 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:18:05.514386 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:18:05.514576 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:18:05.514643 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:05.514645 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:05.515622 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:05.515623 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:05.516310 (+ 687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:05.516313 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:05.516702 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:05.516703 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:05.516782 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:05.517230 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:18:05.517236 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:18:05.517628 (+ 392us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:18:05.519261 (+ 1633us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:18:05.519262 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:18:05.519262 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:18:05.519467 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:18:05.519662 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:18:05.519663 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:18:05.519664 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:18:05.519746 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:18:05.520007 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:18:05.520008 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:18:05.520053 (+ 45us) client_negotiation.cc:769] Sending connection context
0929 03:18:05.520110 (+ 57us) client_negotiation.cc:240] Negotiation successful
0929 03:18:05.520182 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":70,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.20.178.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.20.178.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651852476208-21193-0/krb5kdc/oryx_127.20.178.67.keytab.
Entry for principal oryx/127.20.178.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651852476208-21193-0/krb5kdc/oryx_127.20.178.67.keytab.
I20260929 03:18:05.536098 21193 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuoaHbX/build/release/bin/kudu
/tmp/dist-test-taskuoaHbX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651852476208-21193-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651852476208-21193-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651852476208-21193-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651852476208-21193-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.178.67:0
--local_ip_for_outbound_sockets=127.20.178.67
--webserver_interface=127.20.178.67
--webserver_port=0
--tserver_master_addrs=127.20.178.126:42107
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651852476208-21193-0/krb5kdc/oryx_127.20.178.67.keytab
--principal=oryx/127.20.178.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.178.84:46725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651852476208-21193-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651852476208-21193-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651852476208-21193-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:18:05.617236 29056 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:18:05.617445 29056 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:18:05.617470 29056 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:18:05.618630 29056 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:18:05.618676 29056 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:18:05.618705 29056 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 03:18:05.618762 29056 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.178.67
I20260929 03:18:05.620278 29056 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.178.84:46725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651852476208-21193-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651852476208-21193-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651852476208-21193-0/krb5kdc/oryx_127.20.178.67.keytab
--principal=oryx/127.20.178.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.178.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651852476208-21193-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.178.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.20.178.126:42107
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29056
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.178.67
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651852476208-21193-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:18:05.620540 29056 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:18:05.620810 29056 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:18:05.625649 29061 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:18:05.625715 29056 server_base.cc:1034] running on GCE node
W20260929 03:18:05.625654 29062 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:18:05.625649 29064 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:18:05.626098 29056 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:18:05.626303 29056 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:18:05.627439 29056 hybrid_clock.cc:648] HybridClock initialized: now 1790651885627423 us; error 41 us; skew 500 ppm
Sep 29 03:18:05 dist-test-slave-c9qh krb5kdc[28659](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651885, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.178.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:18:05.629940 29056 init.cc:377] Logged in from keytab as oryx/127.20.178.67@KRBTEST.COM (short username oryx)
I20260929 03:18:05.630393 29056 webserver.cc:460] Webserver started at http://127.20.178.67:37699/ using document root <none> and password file <none>
I20260929 03:18:05.630592 29056 fs_manager.cc:359] Metadata directory not provided
I20260929 03:18:05.630633 29056 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:18:05.630729 29056 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:18:05.631712 29056 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651852476208-21193-0/minicluster-data/ts-2/data/instance:
uuid: "d17b19f91f0c44a48c760e9cbf12de0a"
format_stamp: "Formatted at 2026-09-29 03:18:05 on dist-test-slave-c9qh"
server_key: "1198db1c805448188c6cac6cc7f7437e"
server_key_iv: "ef1c71870a699093fc1012763c4f0ace"
server_key_version: "encryptionkey@0"
I20260929 03:18:05.632005 29056 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651852476208-21193-0/minicluster-data/ts-2/wal/instance:
uuid: "d17b19f91f0c44a48c760e9cbf12de0a"
format_stamp: "Formatted at 2026-09-29 03:18:05 on dist-test-slave-c9qh"
server_key: "1198db1c805448188c6cac6cc7f7437e"
server_key_iv: "ef1c71870a699093fc1012763c4f0ace"
server_key_version: "encryptionkey@0"
I20260929 03:18:05.633334 29056 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 03:18:05.634030 29071 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:18:05.634222 29056 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 03:18:05.634301 29056 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651852476208-21193-0/minicluster-data/ts-2/data,/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651852476208-21193-0/minicluster-data/ts-2/wal
uuid: "d17b19f91f0c44a48c760e9cbf12de0a"
format_stamp: "Formatted at 2026-09-29 03:18:05 on dist-test-slave-c9qh"
server_key: "1198db1c805448188c6cac6cc7f7437e"
server_key_iv: "ef1c71870a699093fc1012763c4f0ace"
server_key_version: "encryptionkey@0"
I20260929 03:18:05.634357 29056 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651852476208-21193-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651852476208-21193-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651852476208-21193-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:18:05.650920 29056 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:18:05.653039 29056 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:18:05.653157 29056 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:18:05.653658 29056 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:18:05.653698 29056 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:18:05.653721 29056 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:18:05.653734 29056 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 03:18:05 dist-test-slave-c9qh krb5kdc[28659](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651885, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.178.67@KRBTEST.COM for oryx/127.20.178.126@KRBTEST.COM
I20260929 03:18:05.661065 29056 rpc_server.cc:307] RPC server started. Bound to: 127.20.178.67:45077
I20260929 03:18:05.661136 29186 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.178.67:45077 every 8 connection(s)
I20260929 03:18:05.661490 29056 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651852476208-21193-0/minicluster-data/ts-2/data/info.pb
I20260929 03:18:05.664037 28773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:05.654147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.67:36443 (local address 127.20.178.126:42107)
0929 03:18:05.654221 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0929 03:18:05.654223 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:18:05.654412 (+ 189us) server_negotiation.cc:407] Connection header received
0929 03:18:05.654990 (+ 578us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:18:05.654992 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:18:05.655008 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:18:05.655118 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:18:05.655988 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:05.656876 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:05.657918 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:05.658271 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:05.658356 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:05.661154 (+ 2798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:18:05.661155 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:18:05.661155 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:18:05.661164 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:18:05.662665 (+ 1501us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:18:05.663033 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:18:05.663034 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:18:05.663034 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:18:05.663083 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:18:05.663343 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:18:05.663344 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:18:05.663344 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:18:05.663465 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:18:05.663530 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0929 03:18:05.663834 (+ 304us) server_negotiation.cc:299] Negotiation successful
0929 03:18:05.663908 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260929 03:18:05.664072 29080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:05.654119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:42107 (local address 127.20.178.67:36443)
0929 03:18:05.654296 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:05.654337 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 03:18:05.654841 (+ 504us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:18:05.655193 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:18:05.655198 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:18:05.655554 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:18:05.655878 (+ 324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:05.655883 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:05.656976 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:05.656977 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:05.657850 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:05.657852 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:05.658374 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:05.658375 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:05.658522 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:05.659158 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:18:05.659175 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:18:05.661032 (+ 1857us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:18:05.662736 (+ 1704us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:18:05.662741 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:18:05.662743 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:18:05.662979 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:18:05.663158 (+ 179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:18:05.663159 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:18:05.663159 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:18:05.663253 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:18:05.663532 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:18:05.663536 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:18:05.663684 (+ 148us) client_negotiation.cc:769] Sending connection context
0929 03:18:05.663788 (+ 104us) client_negotiation.cc:240] Negotiation successful
0929 03:18:05.663915 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":113,"threads_started":1}
I20260929 03:18:05.664598 29187 heartbeater.cc:347] Connected to a master server at 127.20.178.126:42107
I20260929 03:18:05.664706 29187 heartbeater.cc:464] Registering TS with master...
I20260929 03:18:05.664891 29187 heartbeater.cc:511] Master 127.20.178.126:42107 requested a full tablet report, sending...
I20260929 03:18:05.665277 28703 ts_manager.cc:194] Registered new tserver with Master: d17b19f91f0c44a48c760e9cbf12de0a (127.20.178.67:45077)
I20260929 03:18:05.665910 28703 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.20.178.67@KRBTEST.COM'} at 127.20.178.67:36443
I20260929 03:18:05.670789 29193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:05.665653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:42107 (local address 127.20.178.67:51619)
0929 03:18:05.665805 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:05.665814 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 03:18:05.665865 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:18:05.666006 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:18:05.666007 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:18:05.666180 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:18:05.666280 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:05.666282 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:05.666901 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:05.666901 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:05.667382 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:05.667383 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:05.667734 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:05.667735 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:05.667782 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:05.668236 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:18:05.668240 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:18:05.668530 (+ 290us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:18:05.669925 (+ 1395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:18:05.669926 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:18:05.669926 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:18:05.670076 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:18:05.670286 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:18:05.670287 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:18:05.670287 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:18:05.670335 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:18:05.670534 (+ 199us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:18:05.670534 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:18:05.670565 (+ 31us) client_negotiation.cc:769] Sending connection context
0929 03:18:05.670616 (+ 51us) client_negotiation.cc:240] Negotiation successful
0929 03:18:05.670687 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":68,"threads_started":1}
I20260929 03:18:05.670899 28773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:05.665704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.67:51619 (local address 127.20.178.126:42107)
0929 03:18:05.665744 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0929 03:18:05.665746 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:18:05.665876 (+ 130us) server_negotiation.cc:407] Connection header received
0929 03:18:05.665915 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:18:05.665916 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:18:05.665926 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:18:05.666000 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:18:05.666362 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:05.666813 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:05.667472 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:05.667658 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:05.667716 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:05.668688 (+ 972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:18:05.668689 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:18:05.668690 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:18:05.668697 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:18:05.669854 (+ 1157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:18:05.670161 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:18:05.670163 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:18:05.670163 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:18:05.670219 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:18:05.670389 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:18:05.670390 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:18:05.670390 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:18:05.670486 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:18:05.670542 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0929 03:18:05.670723 (+ 181us) server_negotiation.cc:299] Negotiation successful
0929 03:18:05.670803 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260929 03:18:05.670993 21193 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuoaHbX/build/release/bin/kudu as pid 29056
I20260929 03:18:05.671124 21193 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651852476208-21193-0/minicluster-data/ts-2/wal/instance
I20260929 03:18:05.671352 21193 external_mini_cluster.cc:1428] Setting key 3bb2f136aa7e6232a6468646eddd6954
I20260929 03:18:05.672329 21193 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 03:18:05 dist-test-slave-c9qh krb5kdc[28659](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651885, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 03:18:05 dist-test-slave-c9qh krb5kdc[28659](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.20.178.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:18:05 dist-test-slave-c9qh krb5kdc[28659](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.20.178.126@KRBTEST.COM, Server not found in Kerberos database
W20260929 03:18:05.693383 28773 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:18:05.688046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57092 (local address 127.20.178.126:42107)
0929 03:18:05.688139 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0929 03:18:05.688141 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:18:05.688157 (+ 16us) server_negotiation.cc:407] Connection header received
0929 03:18:05.688267 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:18:05.688268 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:18:05.688281 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:18:05.688343 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:18:05.689019 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:05.689495 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:05.690211 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:05.690381 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:05.690429 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:05.693301 (+ 2872us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57092: BlockingRecv error: recv got EOF from 127.0.0.1:57092 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 29 03:18:05 dist-test-slave-c9qh krb5kdc[28659](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651885, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.20.178.126@KRBTEST.COM
I20260929 03:18:05.700505 28773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:05.694156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57098 (local address 127.20.178.126:42107)
0929 03:18:05.694228 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0929 03:18:05.694230 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:18:05.694322 (+ 92us) server_negotiation.cc:407] Connection header received
0929 03:18:05.694420 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:18:05.694421 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:18:05.694430 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:18:05.694476 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:18:05.694911 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:05.695378 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:05.696060 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:05.696251 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:05.696334 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:05.698346 (+ 2012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:18:05.698347 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:18:05.698348 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:18:05.698354 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:18:05.699455 (+ 1101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:18:05.699748 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:18:05.699749 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:18:05.699749 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:18:05.699797 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:18:05.699988 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:18:05.699989 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:18:05.699989 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:18:05.700108 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:18:05.700198 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0929 03:18:05.700328 (+ 130us) server_negotiation.cc:299] Negotiation successful
0929 03:18:05.700404 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 03:18:05.701965 28703 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57098:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 03:18:05.703419 28703 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 03:18:05.709645 29206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:05.705574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.66:44909 (local address 127.0.0.1:33508)
0929 03:18:05.705941 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:05.705978 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 03:18:05.706076 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:18:05.706395 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:18:05.706398 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:18:05.706400 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:18:05.706467 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:05.706469 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:05.707336 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:05.707337 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:05.708466 (+ 1129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:05.708469 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:05.709346 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:05.709347 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:05.709435 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:05.709449 (+ 14us) client_negotiation.cc:769] Sending connection context
0929 03:18:05.709523 (+ 74us) client_negotiation.cc:240] Negotiation successful
0929 03:18:05.709569 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":55,"threads_started":1}
I20260929 03:18:05.709699 29208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:05.705677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33508 (local address 127.20.178.66:44909)
0929 03:18:05.705854 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0929 03:18:05.705858 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:18:05.706062 (+ 204us) server_negotiation.cc:407] Connection header received
0929 03:18:05.706208 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:18:05.706212 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:18:05.706231 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:18:05.706316 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:18:05.706554 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:05.707255 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:05.708571 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:05.709258 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:05.709330 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:05.709398 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0929 03:18:05.709549 (+ 151us) server_negotiation.cc:299] Negotiation successful
0929 03:18:05.709591 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":71,"threads_started":1}
I20260929 03:18:05.710108 29209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:05.705581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.67:45077 (local address 127.0.0.1:38304)
0929 03:18:05.706059 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:05.706069 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 03:18:05.706135 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:18:05.706401 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:18:05.706402 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:18:05.706404 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:18:05.706471 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:05.706473 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:05.707744 (+ 1271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:05.707745 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:05.708882 (+ 1137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:05.708884 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:05.709878 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:05.709879 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:05.709945 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:05.709949 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 03:18:05.710006 (+ 57us) client_negotiation.cc:240] Negotiation successful
0929 03:18:05.710044 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":50,"threads_started":1}
I20260929 03:18:05.710177 29210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:05.705761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38304 (local address 127.20.178.67:45077)
0929 03:18:05.705921 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0929 03:18:05.705925 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:18:05.706136 (+ 211us) server_negotiation.cc:407] Connection header received
0929 03:18:05.706229 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:18:05.706231 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:18:05.706243 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:18:05.706329 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:18:05.706553 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:05.707661 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:05.708964 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:05.709702 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:05.709774 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:05.709845 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0929 03:18:05.710023 (+ 178us) server_negotiation.cc:299] Negotiation successful
0929 03:18:05.710070 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":87,"threads_started":1}
I20260929 03:18:05.710355 28981 tablet_service.cc:1467] Processing CreateTablet for tablet 0fa2bfe60b144281bbfd29f592eadf97 (DEFAULT_TABLE table=test-table [id=ca041dba57e74a659f6b0bc9694221b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:18:05.710721 28981 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0fa2bfe60b144281bbfd29f592eadf97. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:18:05.710851 29121 tablet_service.cc:1467] Processing CreateTablet for tablet 0fa2bfe60b144281bbfd29f592eadf97 (DEFAULT_TABLE table=test-table [id=ca041dba57e74a659f6b0bc9694221b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:18:05.711199 29121 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0fa2bfe60b144281bbfd29f592eadf97. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:18:05.711491 29207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:05.705782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.65:46097 (local address 127.0.0.1:47574)
0929 03:18:05.706230 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:05.706238 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0929 03:18:05.706285 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:18:05.706506 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:18:05.706507 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:18:05.706509 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:18:05.706543 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:05.706544 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:05.708961 (+ 2417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:05.708962 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:05.709990 (+ 1028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:05.709993 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:05.711204 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:05.711205 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:05.711316 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:05.711320 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 03:18:05.711384 (+ 64us) client_negotiation.cc:240] Negotiation successful
0929 03:18:05.711421 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":35,"threads_started":1}
I20260929 03:18:05.711565 29211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:05.705883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47574 (local address 127.20.178.65:46097)
0929 03:18:05.706047 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0929 03:18:05.706052 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:18:05.706308 (+ 256us) server_negotiation.cc:407] Connection header received
0929 03:18:05.706353 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:18:05.706355 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:18:05.706370 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:18:05.706440 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:18:05.706618 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:05.707647 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:05.710081 (+ 2434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:05.711117 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:05.711194 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:05.711268 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0929 03:18:05.711408 (+ 140us) server_negotiation.cc:299] Negotiation successful
0929 03:18:05.711446 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":97,"threads_started":1}
I20260929 03:18:05.712366 28841 tablet_service.cc:1467] Processing CreateTablet for tablet 0fa2bfe60b144281bbfd29f592eadf97 (DEFAULT_TABLE table=test-table [id=ca041dba57e74a659f6b0bc9694221b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:18:05.712769 28841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0fa2bfe60b144281bbfd29f592eadf97. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:18:05.714433 29212 tablet_bootstrap.cc:492] T 0fa2bfe60b144281bbfd29f592eadf97 P d17b19f91f0c44a48c760e9cbf12de0a: Bootstrap starting.
I20260929 03:18:05.714850 29213 tablet_bootstrap.cc:492] T 0fa2bfe60b144281bbfd29f592eadf97 P c8ef706c3c2141aa895ff4a4b0b3a3da: Bootstrap starting.
I20260929 03:18:05.715219 29212 tablet_bootstrap.cc:654] T 0fa2bfe60b144281bbfd29f592eadf97 P d17b19f91f0c44a48c760e9cbf12de0a: Neither blocks nor log segments found. Creating new log.
I20260929 03:18:05.715466 29213 tablet_bootstrap.cc:654] T 0fa2bfe60b144281bbfd29f592eadf97 P c8ef706c3c2141aa895ff4a4b0b3a3da: Neither blocks nor log segments found. Creating new log.
I20260929 03:18:05.715534 29212 log.cc:824] T 0fa2bfe60b144281bbfd29f592eadf97 P d17b19f91f0c44a48c760e9cbf12de0a: Log is configured to *not* fsync() on all Append() calls
I20260929 03:18:05.715787 29213 log.cc:824] T 0fa2bfe60b144281bbfd29f592eadf97 P c8ef706c3c2141aa895ff4a4b0b3a3da: Log is configured to *not* fsync() on all Append() calls
I20260929 03:18:05.716290 29212 tablet_bootstrap.cc:492] T 0fa2bfe60b144281bbfd29f592eadf97 P d17b19f91f0c44a48c760e9cbf12de0a: No bootstrap required, opened a new log
I20260929 03:18:05.716372 29212 ts_tablet_manager.cc:1397] T 0fa2bfe60b144281bbfd29f592eadf97 P d17b19f91f0c44a48c760e9cbf12de0a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 03:18:05.716497 29213 tablet_bootstrap.cc:492] T 0fa2bfe60b144281bbfd29f592eadf97 P c8ef706c3c2141aa895ff4a4b0b3a3da: No bootstrap required, opened a new log
I20260929 03:18:05.716562 29213 ts_tablet_manager.cc:1397] T 0fa2bfe60b144281bbfd29f592eadf97 P c8ef706c3c2141aa895ff4a4b0b3a3da: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 03:18:05.717787 29212 raft_consensus.cc:348] T 0fa2bfe60b144281bbfd29f592eadf97 P d17b19f91f0c44a48c760e9cbf12de0a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8ef706c3c2141aa895ff4a4b0b3a3da" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 44909 } } peers { permanent_uuid: "d17b19f91f0c44a48c760e9cbf12de0a" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 45077 } } peers { permanent_uuid: "8bb9788606a844aeac11ef74e3475047" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 46097 } }
I20260929 03:18:05.717914 29212 raft_consensus.cc:374] T 0fa2bfe60b144281bbfd29f592eadf97 P d17b19f91f0c44a48c760e9cbf12de0a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:18:05.717937 29212 raft_consensus.cc:729] T 0fa2bfe60b144281bbfd29f592eadf97 P d17b19f91f0c44a48c760e9cbf12de0a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d17b19f91f0c44a48c760e9cbf12de0a, State: Initialized, Role: FOLLOWER
I20260929 03:18:05.717877 29213 raft_consensus.cc:348] T 0fa2bfe60b144281bbfd29f592eadf97 P c8ef706c3c2141aa895ff4a4b0b3a3da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8ef706c3c2141aa895ff4a4b0b3a3da" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 44909 } } peers { permanent_uuid: "d17b19f91f0c44a48c760e9cbf12de0a" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 45077 } } peers { permanent_uuid: "8bb9788606a844aeac11ef74e3475047" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 46097 } }
I20260929 03:18:05.718048 29213 raft_consensus.cc:374] T 0fa2bfe60b144281bbfd29f592eadf97 P c8ef706c3c2141aa895ff4a4b0b3a3da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:18:05.718060 29212 consensus_queue.cc:260] T 0fa2bfe60b144281bbfd29f592eadf97 P d17b19f91f0c44a48c760e9cbf12de0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8ef706c3c2141aa895ff4a4b0b3a3da" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 44909 } } peers { permanent_uuid: "d17b19f91f0c44a48c760e9cbf12de0a" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 45077 } } peers { permanent_uuid: "8bb9788606a844aeac11ef74e3475047" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 46097 } }
I20260929 03:18:05.718098 29213 raft_consensus.cc:729] T 0fa2bfe60b144281bbfd29f592eadf97 P c8ef706c3c2141aa895ff4a4b0b3a3da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8ef706c3c2141aa895ff4a4b0b3a3da, State: Initialized, Role: FOLLOWER
I20260929 03:18:05.718215 29213 consensus_queue.cc:260] T 0fa2bfe60b144281bbfd29f592eadf97 P c8ef706c3c2141aa895ff4a4b0b3a3da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8ef706c3c2141aa895ff4a4b0b3a3da" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 44909 } } peers { permanent_uuid: "d17b19f91f0c44a48c760e9cbf12de0a" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 45077 } } peers { permanent_uuid: "8bb9788606a844aeac11ef74e3475047" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 46097 } }
I20260929 03:18:05.718329 29212 ts_tablet_manager.cc:1428] T 0fa2bfe60b144281bbfd29f592eadf97 P d17b19f91f0c44a48c760e9cbf12de0a: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260929 03:18:05.718425 29187 heartbeater.cc:503] Master 127.20.178.126:42107 was elected leader, sending a full tablet report...
I20260929 03:18:05.718497 29047 heartbeater.cc:503] Master 127.20.178.126:42107 was elected leader, sending a full tablet report...
I20260929 03:18:05.718490 29213 ts_tablet_manager.cc:1428] T 0fa2bfe60b144281bbfd29f592eadf97 P c8ef706c3c2141aa895ff4a4b0b3a3da: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260929 03:18:05.718935 29216 tablet_bootstrap.cc:492] T 0fa2bfe60b144281bbfd29f592eadf97 P 8bb9788606a844aeac11ef74e3475047: Bootstrap starting.
I20260929 03:18:05.719759 29216 tablet_bootstrap.cc:654] T 0fa2bfe60b144281bbfd29f592eadf97 P 8bb9788606a844aeac11ef74e3475047: Neither blocks nor log segments found. Creating new log.
I20260929 03:18:05.719991 29216 log.cc:824] T 0fa2bfe60b144281bbfd29f592eadf97 P 8bb9788606a844aeac11ef74e3475047: Log is configured to *not* fsync() on all Append() calls
I20260929 03:18:05.720572 29216 tablet_bootstrap.cc:492] T 0fa2bfe60b144281bbfd29f592eadf97 P 8bb9788606a844aeac11ef74e3475047: No bootstrap required, opened a new log
I20260929 03:18:05.720625 29216 ts_tablet_manager.cc:1397] T 0fa2bfe60b144281bbfd29f592eadf97 P 8bb9788606a844aeac11ef74e3475047: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 03:18:05.721973 29216 raft_consensus.cc:348] T 0fa2bfe60b144281bbfd29f592eadf97 P 8bb9788606a844aeac11ef74e3475047 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8ef706c3c2141aa895ff4a4b0b3a3da" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 44909 } } peers { permanent_uuid: "d17b19f91f0c44a48c760e9cbf12de0a" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 45077 } } peers { permanent_uuid: "8bb9788606a844aeac11ef74e3475047" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 46097 } }
I20260929 03:18:05.722100 29216 raft_consensus.cc:374] T 0fa2bfe60b144281bbfd29f592eadf97 P 8bb9788606a844aeac11ef74e3475047 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:18:05.722122 29216 raft_consensus.cc:729] T 0fa2bfe60b144281bbfd29f592eadf97 P 8bb9788606a844aeac11ef74e3475047 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8bb9788606a844aeac11ef74e3475047, State: Initialized, Role: FOLLOWER
I20260929 03:18:05.722204 29216 consensus_queue.cc:260] T 0fa2bfe60b144281bbfd29f592eadf97 P 8bb9788606a844aeac11ef74e3475047 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8ef706c3c2141aa895ff4a4b0b3a3da" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 44909 } } peers { permanent_uuid: "d17b19f91f0c44a48c760e9cbf12de0a" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 45077 } } peers { permanent_uuid: "8bb9788606a844aeac11ef74e3475047" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 46097 } }
I20260929 03:18:05.722440 29216 ts_tablet_manager.cc:1428] T 0fa2bfe60b144281bbfd29f592eadf97 P 8bb9788606a844aeac11ef74e3475047: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260929 03:18:05.722519 28907 heartbeater.cc:503] Master 127.20.178.126:42107 was elected leader, sending a full tablet report...
W20260929 03:18:05.760154 29048 tablet.cc:2369] T 0fa2bfe60b144281bbfd29f592eadf97 P c8ef706c3c2141aa895ff4a4b0b3a3da: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 03:18:05.854492 28908 tablet.cc:2369] T 0fa2bfe60b144281bbfd29f592eadf97 P 8bb9788606a844aeac11ef74e3475047: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 03:18:05.912206 29188 tablet.cc:2369] T 0fa2bfe60b144281bbfd29f592eadf97 P d17b19f91f0c44a48c760e9cbf12de0a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 03:18:05.949139 29217 raft_consensus.cc:482] T 0fa2bfe60b144281bbfd29f592eadf97 P d17b19f91f0c44a48c760e9cbf12de0a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:18:05.949249 29217 raft_consensus.cc:504] T 0fa2bfe60b144281bbfd29f592eadf97 P d17b19f91f0c44a48c760e9cbf12de0a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8ef706c3c2141aa895ff4a4b0b3a3da" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 44909 } } peers { permanent_uuid: "d17b19f91f0c44a48c760e9cbf12de0a" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 45077 } } peers { permanent_uuid: "8bb9788606a844aeac11ef74e3475047" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 46097 } }
I20260929 03:18:05.949548 29217 leader_election.cc:288] T 0fa2bfe60b144281bbfd29f592eadf97 P d17b19f91f0c44a48c760e9cbf12de0a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c8ef706c3c2141aa895ff4a4b0b3a3da (127.20.178.66:44909), 8bb9788606a844aeac11ef74e3475047 (127.20.178.65:46097)
I20260929 03:18:05.953639 29208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:05.949836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.67:48665 (local address 127.20.178.66:44909)
0929 03:18:05.949892 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0929 03:18:05.949895 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:18:05.949909 (+ 14us) server_negotiation.cc:407] Connection header received
0929 03:18:05.949962 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:18:05.949963 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:18:05.949973 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:18:05.950086 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:18:05.950301 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:05.951037 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:05.952668 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:05.953246 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:05.953313 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:05.953321 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0929 03:18:05.953520 (+ 199us) server_negotiation.cc:299] Negotiation successful
0929 03:18:05.953570 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260929 03:18:05.953691 29211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:05.949836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.67:51121 (local address 127.20.178.65:46097)
0929 03:18:05.949905 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0929 03:18:05.949908 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:18:05.950024 (+ 116us) server_negotiation.cc:407] Connection header received
0929 03:18:05.950052 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:18:05.950053 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:18:05.950063 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:18:05.950118 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:18:05.950301 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:05.951075 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:05.952673 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:05.953220 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:05.953306 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:05.953315 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0929 03:18:05.953520 (+ 205us) server_negotiation.cc:299] Negotiation successful
0929 03:18:05.953569 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 03:18:05.953634 29221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:05.949756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.65:46097 (local address 127.20.178.67:51121)
0929 03:18:05.949945 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:05.949954 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 03:18:05.950007 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:18:05.950132 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:18:05.950132 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:18:05.950133 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:18:05.950218 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:05.950220 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:05.951158 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:05.951159 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:05.952511 (+ 1352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:05.952515 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:05.953318 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:05.953319 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:05.953408 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:05.953415 (+ 7us) client_negotiation.cc:769] Sending connection context
0929 03:18:05.953476 (+ 61us) client_negotiation.cc:240] Negotiation successful
0929 03:18:05.953523 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"mutex_wait_us":100,"thread_start_us":97,"threads_started":1}
I20260929 03:18:05.954047 28859 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fa2bfe60b144281bbfd29f592eadf97" candidate_uuid: "d17b19f91f0c44a48c760e9cbf12de0a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8bb9788606a844aeac11ef74e3475047" is_pre_election: true
I20260929 03:18:05.954223 28859 raft_consensus.cc:2454] T 0fa2bfe60b144281bbfd29f592eadf97 P 8bb9788606a844aeac11ef74e3475047 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d17b19f91f0c44a48c760e9cbf12de0a in term 0.
I20260929 03:18:05.953639 29080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:05.949756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.66:44909 (local address 127.20.178.67:48665)
0929 03:18:05.949817 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:05.949829 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 03:18:05.949883 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:18:05.950114 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:18:05.950115 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:18:05.950117 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:18:05.950218 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:05.950220 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:05.951158 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:05.951159 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:05.952511 (+ 1352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:05.952515 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:05.953318 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:05.953319 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:05.953408 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:05.953415 (+ 7us) client_negotiation.cc:769] Sending connection context
0929 03:18:05.953476 (+ 61us) client_negotiation.cc:240] Negotiation successful
0929 03:18:05.953530 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41,"mutex_wait_us":89}
I20260929 03:18:05.954515 29072 leader_election.cc:302] T 0fa2bfe60b144281bbfd29f592eadf97 P d17b19f91f0c44a48c760e9cbf12de0a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8bb9788606a844aeac11ef74e3475047, d17b19f91f0c44a48c760e9cbf12de0a; no voters:
I20260929 03:18:05.954592 29001 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fa2bfe60b144281bbfd29f592eadf97" candidate_uuid: "d17b19f91f0c44a48c760e9cbf12de0a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8ef706c3c2141aa895ff4a4b0b3a3da" is_pre_election: true
I20260929 03:18:05.954674 29217 raft_consensus.cc:2781] T 0fa2bfe60b144281bbfd29f592eadf97 P d17b19f91f0c44a48c760e9cbf12de0a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 03:18:05.954721 29217 raft_consensus.cc:482] T 0fa2bfe60b144281bbfd29f592eadf97 P d17b19f91f0c44a48c760e9cbf12de0a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 03:18:05.954742 29217 raft_consensus.cc:3037] T 0fa2bfe60b144281bbfd29f592eadf97 P d17b19f91f0c44a48c760e9cbf12de0a [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:18:05.954741 29001 raft_consensus.cc:2454] T 0fa2bfe60b144281bbfd29f592eadf97 P c8ef706c3c2141aa895ff4a4b0b3a3da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d17b19f91f0c44a48c760e9cbf12de0a in term 0.
I20260929 03:18:05.955729 29217 raft_consensus.cc:504] T 0fa2bfe60b144281bbfd29f592eadf97 P d17b19f91f0c44a48c760e9cbf12de0a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8ef706c3c2141aa895ff4a4b0b3a3da" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 44909 } } peers { permanent_uuid: "d17b19f91f0c44a48c760e9cbf12de0a" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 45077 } } peers { permanent_uuid: "8bb9788606a844aeac11ef74e3475047" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 46097 } }
I20260929 03:18:05.955888 29217 leader_election.cc:288] T 0fa2bfe60b144281bbfd29f592eadf97 P d17b19f91f0c44a48c760e9cbf12de0a [CANDIDATE]: Term 1 election: Requested vote from peers c8ef706c3c2141aa895ff4a4b0b3a3da (127.20.178.66:44909), 8bb9788606a844aeac11ef74e3475047 (127.20.178.65:46097)
I20260929 03:18:05.955998 29001 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fa2bfe60b144281bbfd29f592eadf97" candidate_uuid: "d17b19f91f0c44a48c760e9cbf12de0a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8ef706c3c2141aa895ff4a4b0b3a3da"
I20260929 03:18:05.956014 28859 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fa2bfe60b144281bbfd29f592eadf97" candidate_uuid: "d17b19f91f0c44a48c760e9cbf12de0a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8bb9788606a844aeac11ef74e3475047"
I20260929 03:18:05.956079 29001 raft_consensus.cc:3037] T 0fa2bfe60b144281bbfd29f592eadf97 P c8ef706c3c2141aa895ff4a4b0b3a3da [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:18:05.956123 28859 raft_consensus.cc:3037] T 0fa2bfe60b144281bbfd29f592eadf97 P 8bb9788606a844aeac11ef74e3475047 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:18:05.956867 29001 raft_consensus.cc:2454] T 0fa2bfe60b144281bbfd29f592eadf97 P c8ef706c3c2141aa895ff4a4b0b3a3da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d17b19f91f0c44a48c760e9cbf12de0a in term 1.
I20260929 03:18:05.956892 28859 raft_consensus.cc:2454] T 0fa2bfe60b144281bbfd29f592eadf97 P 8bb9788606a844aeac11ef74e3475047 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d17b19f91f0c44a48c760e9cbf12de0a in term 1.
I20260929 03:18:05.957044 29073 leader_election.cc:302] T 0fa2bfe60b144281bbfd29f592eadf97 P d17b19f91f0c44a48c760e9cbf12de0a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c8ef706c3c2141aa895ff4a4b0b3a3da, d17b19f91f0c44a48c760e9cbf12de0a; no voters:
I20260929 03:18:05.957150 29217 raft_consensus.cc:2781] T 0fa2bfe60b144281bbfd29f592eadf97 P d17b19f91f0c44a48c760e9cbf12de0a [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:18:05.957314 29217 raft_consensus.cc:686] T 0fa2bfe60b144281bbfd29f592eadf97 P d17b19f91f0c44a48c760e9cbf12de0a [term 1 LEADER]: Becoming Leader. State: Replica: d17b19f91f0c44a48c760e9cbf12de0a, State: Running, Role: LEADER
I20260929 03:18:05.957427 29217 consensus_queue.cc:237] T 0fa2bfe60b144281bbfd29f592eadf97 P d17b19f91f0c44a48c760e9cbf12de0a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8ef706c3c2141aa895ff4a4b0b3a3da" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 44909 } } peers { permanent_uuid: "d17b19f91f0c44a48c760e9cbf12de0a" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 45077 } } peers { permanent_uuid: "8bb9788606a844aeac11ef74e3475047" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 46097 } }
I20260929 03:18:05.958266 28703 catalog_manager.cc:5495] T 0fa2bfe60b144281bbfd29f592eadf97 P d17b19f91f0c44a48c760e9cbf12de0a reported cstate change: term changed from 0 to 1, leader changed from <none> to d17b19f91f0c44a48c760e9cbf12de0a (127.20.178.67). New cstate: current_term: 1 leader_uuid: "d17b19f91f0c44a48c760e9cbf12de0a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8ef706c3c2141aa895ff4a4b0b3a3da" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 44909 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d17b19f91f0c44a48c760e9cbf12de0a" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 45077 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8bb9788606a844aeac11ef74e3475047" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 46097 } health_report { overall_health: UNKNOWN } } }
I20260929 03:18:05.996852 29210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:05.994313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38308 (local address 127.20.178.67:45077)
0929 03:18:05.994370 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0929 03:18:05.994372 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:18:05.994382 (+ 10us) server_negotiation.cc:407] Connection header received
0929 03:18:05.994469 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:18:05.994470 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:18:05.994487 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:18:05.994578 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:18:05.994749 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:05.995230 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:05.995999 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:05.996239 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:05.996320 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:05.996485 (+ 165us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:18:05.996588 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:18:05.996719 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0929 03:18:05.996739 (+ 20us) server_negotiation.cc:299] Negotiation successful
0929 03:18:05.996759 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260929 03:18:05.998553 29001 raft_consensus.cc:1261] T 0fa2bfe60b144281bbfd29f592eadf97 P c8ef706c3c2141aa895ff4a4b0b3a3da [term 1 FOLLOWER]: Refusing update from remote peer d17b19f91f0c44a48c760e9cbf12de0a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:18:05.998597 28859 raft_consensus.cc:1261] T 0fa2bfe60b144281bbfd29f592eadf97 P 8bb9788606a844aeac11ef74e3475047 [term 1 FOLLOWER]: Refusing update from remote peer d17b19f91f0c44a48c760e9cbf12de0a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:18:05.998900 29217 consensus_queue.cc:1035] T 0fa2bfe60b144281bbfd29f592eadf97 P d17b19f91f0c44a48c760e9cbf12de0a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8ef706c3c2141aa895ff4a4b0b3a3da" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 44909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:18:05.999034 29222 consensus_queue.cc:1035] T 0fa2bfe60b144281bbfd29f592eadf97 P d17b19f91f0c44a48c760e9cbf12de0a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8bb9788606a844aeac11ef74e3475047" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 46097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:18:06.001119 29225 mvcc.cc:205] Tried to move back new op lower bound from 7334510125047427072 to 7334510124882100224. Current Snapshot: MvccSnapshot[applied={T|T < 7334510125047427072}]
I20260929 03:18:06.001395 29226 mvcc.cc:205] Tried to move back new op lower bound from 7334510125047427072 to 7334510124882100224. Current Snapshot: MvccSnapshot[applied={T|T < 7334510125047427072}]
I20260929 03:18:06.002393 29227 mvcc.cc:205] Tried to move back new op lower bound from 7334510125047427072 to 7334510124882100224. Current Snapshot: MvccSnapshot[applied={T|T < 7334510125047427072}]
I20260929 03:18:06.003705 28703 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57098:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:18:06.003799 28703 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57098:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:18:06.004595 28703 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b8fefe3633e6491689c51b85ebdc7336: Sending DeleteTablet for 3 replicas of tablet 0fa2bfe60b144281bbfd29f592eadf97
I20260929 03:18:06.004922 28981 tablet_service.cc:1514] Processing DeleteTablet for tablet 0fa2bfe60b144281bbfd29f592eadf97 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:18:05 UTC) from {username='oryx', principal='oryx/127.20.178.126@KRBTEST.COM'} at 127.0.0.1:33508
I20260929 03:18:06.005191 29121 tablet_service.cc:1514] Processing DeleteTablet for tablet 0fa2bfe60b144281bbfd29f592eadf97 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:18:05 UTC) from {username='oryx', principal='oryx/127.20.178.126@KRBTEST.COM'} at 127.0.0.1:38304
I20260929 03:18:06.005146 28703 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57098:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260929 03:18:06.005245 28841 tablet_service.cc:1514] Processing DeleteTablet for tablet 0fa2bfe60b144281bbfd29f592eadf97 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:18:05 UTC) from {username='oryx', principal='oryx/127.20.178.126@KRBTEST.COM'} at 127.0.0.1:47574
W20260929 03:18:06.005287 28703 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 03:18:06.005652 29234 tablet_replica.cc:330] T 0fa2bfe60b144281bbfd29f592eadf97 P c8ef706c3c2141aa895ff4a4b0b3a3da: stopping tablet replica
I20260929 03:18:06.005748 29234 raft_consensus.cc:2229] T 0fa2bfe60b144281bbfd29f592eadf97 P c8ef706c3c2141aa895ff4a4b0b3a3da [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:18:06.005847 29234 raft_consensus.cc:2258] T 0fa2bfe60b144281bbfd29f592eadf97 P c8ef706c3c2141aa895ff4a4b0b3a3da [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:18:06.006273 29234 ts_tablet_manager.cc:1905] T 0fa2bfe60b144281bbfd29f592eadf97 P c8ef706c3c2141aa895ff4a4b0b3a3da: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:18:06.006445 29121 tablet_service.cc:1467] Processing CreateTablet for tablet 5557c818bab749349ba412c7d6cd73d2 (DEFAULT_TABLE table=test-table [id=fe0b77c11a034e00a99aeb971460b18f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:18:06.006476 28981 tablet_service.cc:1467] Processing CreateTablet for tablet 5557c818bab749349ba412c7d6cd73d2 (DEFAULT_TABLE table=test-table [id=fe0b77c11a034e00a99aeb971460b18f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:18:06.006559 29121 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5557c818bab749349ba412c7d6cd73d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:18:06.006598 28981 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5557c818bab749349ba412c7d6cd73d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:18:06.007573 29213 tablet_bootstrap.cc:492] T 5557c818bab749349ba412c7d6cd73d2 P c8ef706c3c2141aa895ff4a4b0b3a3da: Bootstrap starting.
I20260929 03:18:06.007884 28841 tablet_service.cc:1467] Processing CreateTablet for tablet 5557c818bab749349ba412c7d6cd73d2 (DEFAULT_TABLE table=test-table [id=fe0b77c11a034e00a99aeb971460b18f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:18:06.008003 28841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5557c818bab749349ba412c7d6cd73d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:18:06.008009 29213 tablet_bootstrap.cc:654] T 5557c818bab749349ba412c7d6cd73d2 P c8ef706c3c2141aa895ff4a4b0b3a3da: Neither blocks nor log segments found. Creating new log.
I20260929 03:18:06.008563 29213 tablet_bootstrap.cc:492] T 5557c818bab749349ba412c7d6cd73d2 P c8ef706c3c2141aa895ff4a4b0b3a3da: No bootstrap required, opened a new log
I20260929 03:18:06.008627 29213 ts_tablet_manager.cc:1397] T 5557c818bab749349ba412c7d6cd73d2 P c8ef706c3c2141aa895ff4a4b0b3a3da: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260929 03:18:06.008805 29213 raft_consensus.cc:348] T 5557c818bab749349ba412c7d6cd73d2 P c8ef706c3c2141aa895ff4a4b0b3a3da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bb9788606a844aeac11ef74e3475047" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 46097 } } peers { permanent_uuid: "c8ef706c3c2141aa895ff4a4b0b3a3da" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 44909 } } peers { permanent_uuid: "d17b19f91f0c44a48c760e9cbf12de0a" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 45077 } }
I20260929 03:18:06.008895 29213 raft_consensus.cc:374] T 5557c818bab749349ba412c7d6cd73d2 P c8ef706c3c2141aa895ff4a4b0b3a3da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:18:06.008925 29213 raft_consensus.cc:729] T 5557c818bab749349ba412c7d6cd73d2 P c8ef706c3c2141aa895ff4a4b0b3a3da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8ef706c3c2141aa895ff4a4b0b3a3da, State: Initialized, Role: FOLLOWER
I20260929 03:18:06.008965 29213 consensus_queue.cc:260] T 5557c818bab749349ba412c7d6cd73d2 P c8ef706c3c2141aa895ff4a4b0b3a3da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bb9788606a844aeac11ef74e3475047" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 46097 } } peers { permanent_uuid: "c8ef706c3c2141aa895ff4a4b0b3a3da" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 44909 } } peers { permanent_uuid: "d17b19f91f0c44a48c760e9cbf12de0a" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 45077 } }
I20260929 03:18:06.009068 29213 ts_tablet_manager.cc:1428] T 5557c818bab749349ba412c7d6cd73d2 P c8ef706c3c2141aa895ff4a4b0b3a3da: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260929 03:18:06.009322 29212 tablet_bootstrap.cc:492] T 5557c818bab749349ba412c7d6cd73d2 P d17b19f91f0c44a48c760e9cbf12de0a: Bootstrap starting.
I20260929 03:18:06.009769 29212 tablet_bootstrap.cc:654] T 5557c818bab749349ba412c7d6cd73d2 P d17b19f91f0c44a48c760e9cbf12de0a: Neither blocks nor log segments found. Creating new log.
I20260929 03:18:06.010129 29216 tablet_bootstrap.cc:492] T 5557c818bab749349ba412c7d6cd73d2 P 8bb9788606a844aeac11ef74e3475047: Bootstrap starting.
I20260929 03:18:06.010496 29216 tablet_bootstrap.cc:654] T 5557c818bab749349ba412c7d6cd73d2 P 8bb9788606a844aeac11ef74e3475047: Neither blocks nor log segments found. Creating new log.
I20260929 03:18:06.011137 29212 tablet_bootstrap.cc:492] T 5557c818bab749349ba412c7d6cd73d2 P d17b19f91f0c44a48c760e9cbf12de0a: No bootstrap required, opened a new log
I20260929 03:18:06.011212 29212 ts_tablet_manager.cc:1397] T 5557c818bab749349ba412c7d6cd73d2 P d17b19f91f0c44a48c760e9cbf12de0a: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260929 03:18:06.011355 29212 raft_consensus.cc:348] T 5557c818bab749349ba412c7d6cd73d2 P d17b19f91f0c44a48c760e9cbf12de0a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bb9788606a844aeac11ef74e3475047" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 46097 } } peers { permanent_uuid: "c8ef706c3c2141aa895ff4a4b0b3a3da" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 44909 } } peers { permanent_uuid: "d17b19f91f0c44a48c760e9cbf12de0a" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 45077 } }
I20260929 03:18:06.011422 29212 raft_consensus.cc:374] T 5557c818bab749349ba412c7d6cd73d2 P d17b19f91f0c44a48c760e9cbf12de0a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:18:06.011444 29212 raft_consensus.cc:729] T 5557c818bab749349ba412c7d6cd73d2 P d17b19f91f0c44a48c760e9cbf12de0a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d17b19f91f0c44a48c760e9cbf12de0a, State: Initialized, Role: FOLLOWER
I20260929 03:18:06.011484 29212 consensus_queue.cc:260] T 5557c818bab749349ba412c7d6cd73d2 P d17b19f91f0c44a48c760e9cbf12de0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bb9788606a844aeac11ef74e3475047" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 46097 } } peers { permanent_uuid: "c8ef706c3c2141aa895ff4a4b0b3a3da" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 44909 } } peers { permanent_uuid: "d17b19f91f0c44a48c760e9cbf12de0a" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 45077 } }
I20260929 03:18:06.011575 29212 ts_tablet_manager.cc:1428] T 5557c818bab749349ba412c7d6cd73d2 P d17b19f91f0c44a48c760e9cbf12de0a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260929 03:18:06.011806 29216 tablet_bootstrap.cc:492] T 5557c818bab749349ba412c7d6cd73d2 P 8bb9788606a844aeac11ef74e3475047: No bootstrap required, opened a new log
I20260929 03:18:06.011860 29216 ts_tablet_manager.cc:1397] T 5557c818bab749349ba412c7d6cd73d2 P 8bb9788606a844aeac11ef74e3475047: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260929 03:18:06.012012 29216 raft_consensus.cc:348] T 5557c818bab749349ba412c7d6cd73d2 P 8bb9788606a844aeac11ef74e3475047 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bb9788606a844aeac11ef74e3475047" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 46097 } } peers { permanent_uuid: "c8ef706c3c2141aa895ff4a4b0b3a3da" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 44909 } } peers { permanent_uuid: "d17b19f91f0c44a48c760e9cbf12de0a" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 45077 } }
I20260929 03:18:06.012082 29216 raft_consensus.cc:374] T 5557c818bab749349ba412c7d6cd73d2 P 8bb9788606a844aeac11ef74e3475047 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:18:06.012111 29216 raft_consensus.cc:729] T 5557c818bab749349ba412c7d6cd73d2 P 8bb9788606a844aeac11ef74e3475047 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8bb9788606a844aeac11ef74e3475047, State: Initialized, Role: FOLLOWER
I20260929 03:18:06.012149 29216 consensus_queue.cc:260] T 5557c818bab749349ba412c7d6cd73d2 P 8bb9788606a844aeac11ef74e3475047 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bb9788606a844aeac11ef74e3475047" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 46097 } } peers { permanent_uuid: "c8ef706c3c2141aa895ff4a4b0b3a3da" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 44909 } } peers { permanent_uuid: "d17b19f91f0c44a48c760e9cbf12de0a" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 45077 } }
I20260929 03:18:06.012250 29216 ts_tablet_manager.cc:1428] T 5557c818bab749349ba412c7d6cd73d2 P 8bb9788606a844aeac11ef74e3475047: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260929 03:18:06.012831 29235 tablet_replica.cc:330] T 0fa2bfe60b144281bbfd29f592eadf97 P d17b19f91f0c44a48c760e9cbf12de0a: stopping tablet replica
I20260929 03:18:06.012948 29235 raft_consensus.cc:2229] T 0fa2bfe60b144281bbfd29f592eadf97 P d17b19f91f0c44a48c760e9cbf12de0a [term 1 LEADER]: Raft consensus shutting down.
I20260929 03:18:06.013067 29235 raft_consensus.cc:2258] T 0fa2bfe60b144281bbfd29f592eadf97 P d17b19f91f0c44a48c760e9cbf12de0a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:18:06.013222 29236 tablet_replica.cc:330] T 0fa2bfe60b144281bbfd29f592eadf97 P 8bb9788606a844aeac11ef74e3475047: stopping tablet replica
I20260929 03:18:06.013339 29236 raft_consensus.cc:2229] T 0fa2bfe60b144281bbfd29f592eadf97 P 8bb9788606a844aeac11ef74e3475047 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:18:06.013451 29236 raft_consensus.cc:2258] T 0fa2bfe60b144281bbfd29f592eadf97 P 8bb9788606a844aeac11ef74e3475047 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:18:06.013952 29236 ts_tablet_manager.cc:1905] T 0fa2bfe60b144281bbfd29f592eadf97 P 8bb9788606a844aeac11ef74e3475047: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:18:06.013988 29235 ts_tablet_manager.cc:1905] T 0fa2bfe60b144281bbfd29f592eadf97 P d17b19f91f0c44a48c760e9cbf12de0a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:18:06.014145 29234 ts_tablet_manager.cc:1918] T 0fa2bfe60b144281bbfd29f592eadf97 P c8ef706c3c2141aa895ff4a4b0b3a3da: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:18:06.014218 29234 log.cc:1196] T 0fa2bfe60b144281bbfd29f592eadf97 P c8ef706c3c2141aa895ff4a4b0b3a3da: Deleting WAL directory at /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651852476208-21193-0/minicluster-data/ts-1/wal/wals/0fa2bfe60b144281bbfd29f592eadf97
I20260929 03:18:06.014467 29234 ts_tablet_manager.cc:1939] T 0fa2bfe60b144281bbfd29f592eadf97 P c8ef706c3c2141aa895ff4a4b0b3a3da: Deleting consensus metadata
I20260929 03:18:06.014991 28689 catalog_manager.cc:4841] TS c8ef706c3c2141aa895ff4a4b0b3a3da (127.20.178.66:44909): tablet 0fa2bfe60b144281bbfd29f592eadf97 (table test-table [id=ca041dba57e74a659f6b0bc9694221b0]) successfully deleted
I20260929 03:18:06.015585 29236 ts_tablet_manager.cc:1918] T 0fa2bfe60b144281bbfd29f592eadf97 P 8bb9788606a844aeac11ef74e3475047: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:18:06.015578 29235 ts_tablet_manager.cc:1918] T 0fa2bfe60b144281bbfd29f592eadf97 P d17b19f91f0c44a48c760e9cbf12de0a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:18:06.015654 29235 log.cc:1196] T 0fa2bfe60b144281bbfd29f592eadf97 P d17b19f91f0c44a48c760e9cbf12de0a: Deleting WAL directory at /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651852476208-21193-0/minicluster-data/ts-2/wal/wals/0fa2bfe60b144281bbfd29f592eadf97
I20260929 03:18:06.015654 29236 log.cc:1196] T 0fa2bfe60b144281bbfd29f592eadf97 P 8bb9788606a844aeac11ef74e3475047: Deleting WAL directory at /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651852476208-21193-0/minicluster-data/ts-0/wal/wals/0fa2bfe60b144281bbfd29f592eadf97
I20260929 03:18:06.015916 29235 ts_tablet_manager.cc:1939] T 0fa2bfe60b144281bbfd29f592eadf97 P d17b19f91f0c44a48c760e9cbf12de0a: Deleting consensus metadata
I20260929 03:18:06.015916 29236 ts_tablet_manager.cc:1939] T 0fa2bfe60b144281bbfd29f592eadf97 P 8bb9788606a844aeac11ef74e3475047: Deleting consensus metadata
I20260929 03:18:06.016324 28688 catalog_manager.cc:4841] TS 8bb9788606a844aeac11ef74e3475047 (127.20.178.65:46097): tablet 0fa2bfe60b144281bbfd29f592eadf97 (table test-table [id=ca041dba57e74a659f6b0bc9694221b0]) successfully deleted
I20260929 03:18:06.016412 28688 catalog_manager.cc:4841] TS d17b19f91f0c44a48c760e9cbf12de0a (127.20.178.67:45077): tablet 0fa2bfe60b144281bbfd29f592eadf97 (table test-table [id=ca041dba57e74a659f6b0bc9694221b0]) successfully deleted
I20260929 03:18:06.228960 29218 raft_consensus.cc:482] T 5557c818bab749349ba412c7d6cd73d2 P c8ef706c3c2141aa895ff4a4b0b3a3da [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:18:06.229081 29218 raft_consensus.cc:504] T 5557c818bab749349ba412c7d6cd73d2 P c8ef706c3c2141aa895ff4a4b0b3a3da [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bb9788606a844aeac11ef74e3475047" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 46097 } } peers { permanent_uuid: "c8ef706c3c2141aa895ff4a4b0b3a3da" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 44909 } } peers { permanent_uuid: "d17b19f91f0c44a48c760e9cbf12de0a" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 45077 } }
I20260929 03:18:06.229501 29218 leader_election.cc:288] T 5557c818bab749349ba412c7d6cd73d2 P c8ef706c3c2141aa895ff4a4b0b3a3da [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8bb9788606a844aeac11ef74e3475047 (127.20.178.65:46097), d17b19f91f0c44a48c760e9cbf12de0a (127.20.178.67:45077)
I20260929 03:18:06.232836 29210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:06.230057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.66:57251 (local address 127.20.178.67:45077)
0929 03:18:06.230108 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0929 03:18:06.230110 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:18:06.230235 (+ 125us) server_negotiation.cc:407] Connection header received
0929 03:18:06.230264 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:18:06.230264 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:18:06.230276 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:18:06.230335 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:18:06.230500 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:06.230937 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:06.232075 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:06.232446 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:06.232502 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:06.232508 (+ 6us) server_negotiation.cc:1035] Waiting for connection context
0929 03:18:06.232721 (+ 213us) server_negotiation.cc:299] Negotiation successful
0929 03:18:06.232768 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260929 03:18:06.232856 29211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:06.229891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.66:51675 (local address 127.20.178.65:46097)
0929 03:18:06.229978 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0929 03:18:06.229981 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:18:06.230059 (+ 78us) server_negotiation.cc:407] Connection header received
0929 03:18:06.230170 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:18:06.230171 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:18:06.230182 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:18:06.230238 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:18:06.230419 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:06.230933 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:06.232117 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:06.232462 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:06.232565 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:06.232573 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0929 03:18:06.232760 (+ 187us) server_negotiation.cc:299] Negotiation successful
0929 03:18:06.232794 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 03:18:06.232857 29241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:06.229994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.67:45077 (local address 127.20.178.66:57251)
0929 03:18:06.230138 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:06.230149 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 03:18:06.230214 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:18:06.230347 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:18:06.230348 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:18:06.230350 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:18:06.230420 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:06.230421 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:06.231176 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:06.231177 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:06.231973 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:06.231975 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:06.232524 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:06.232525 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:06.232607 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:06.232611 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 03:18:06.232717 (+ 106us) client_negotiation.cc:240] Negotiation successful
0929 03:18:06.232767 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":56,"threads_started":1}
I20260929 03:18:06.233127 29240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:06.229790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.65:46097 (local address 127.20.178.66:51675)
0929 03:18:06.229973 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:06.229987 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 03:18:06.230076 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:18:06.230257 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:18:06.230258 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:18:06.230261 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:18:06.230341 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:06.230343 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:06.231112 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:06.231113 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:06.231978 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:06.231980 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:06.232578 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:06.232579 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:06.232689 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:06.232692 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 03:18:06.232752 (+ 60us) client_negotiation.cc:240] Negotiation successful
0929 03:18:06.232792 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"mutex_wait_us":24,"thread_start_us":76,"threads_started":1}
I20260929 03:18:06.233186 29136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5557c818bab749349ba412c7d6cd73d2" candidate_uuid: "c8ef706c3c2141aa895ff4a4b0b3a3da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d17b19f91f0c44a48c760e9cbf12de0a" is_pre_election: true
I20260929 03:18:06.233333 29136 raft_consensus.cc:2454] T 5557c818bab749349ba412c7d6cd73d2 P d17b19f91f0c44a48c760e9cbf12de0a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8ef706c3c2141aa895ff4a4b0b3a3da in term 0.
I20260929 03:18:06.233563 28932 leader_election.cc:302] T 5557c818bab749349ba412c7d6cd73d2 P c8ef706c3c2141aa895ff4a4b0b3a3da [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c8ef706c3c2141aa895ff4a4b0b3a3da, d17b19f91f0c44a48c760e9cbf12de0a; no voters:
I20260929 03:18:06.233611 28859 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5557c818bab749349ba412c7d6cd73d2" candidate_uuid: "c8ef706c3c2141aa895ff4a4b0b3a3da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8bb9788606a844aeac11ef74e3475047" is_pre_election: true
I20260929 03:18:06.233677 28859 raft_consensus.cc:2454] T 5557c818bab749349ba412c7d6cd73d2 P 8bb9788606a844aeac11ef74e3475047 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8ef706c3c2141aa895ff4a4b0b3a3da in term 0.
I20260929 03:18:06.233736 29218 raft_consensus.cc:2781] T 5557c818bab749349ba412c7d6cd73d2 P c8ef706c3c2141aa895ff4a4b0b3a3da [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 03:18:06.233785 29218 raft_consensus.cc:482] T 5557c818bab749349ba412c7d6cd73d2 P c8ef706c3c2141aa895ff4a4b0b3a3da [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 03:18:06.233805 29218 raft_consensus.cc:3037] T 5557c818bab749349ba412c7d6cd73d2 P c8ef706c3c2141aa895ff4a4b0b3a3da [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:18:06.234616 29218 raft_consensus.cc:504] T 5557c818bab749349ba412c7d6cd73d2 P c8ef706c3c2141aa895ff4a4b0b3a3da [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bb9788606a844aeac11ef74e3475047" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 46097 } } peers { permanent_uuid: "c8ef706c3c2141aa895ff4a4b0b3a3da" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 44909 } } peers { permanent_uuid: "d17b19f91f0c44a48c760e9cbf12de0a" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 45077 } }
I20260929 03:18:06.234716 29218 leader_election.cc:288] T 5557c818bab749349ba412c7d6cd73d2 P c8ef706c3c2141aa895ff4a4b0b3a3da [CANDIDATE]: Term 1 election: Requested vote from peers 8bb9788606a844aeac11ef74e3475047 (127.20.178.65:46097), d17b19f91f0c44a48c760e9cbf12de0a (127.20.178.67:45077)
I20260929 03:18:06.234860 28859 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5557c818bab749349ba412c7d6cd73d2" candidate_uuid: "c8ef706c3c2141aa895ff4a4b0b3a3da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8bb9788606a844aeac11ef74e3475047"
I20260929 03:18:06.234930 29136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5557c818bab749349ba412c7d6cd73d2" candidate_uuid: "c8ef706c3c2141aa895ff4a4b0b3a3da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d17b19f91f0c44a48c760e9cbf12de0a"
I20260929 03:18:06.234972 28859 raft_consensus.cc:3037] T 5557c818bab749349ba412c7d6cd73d2 P 8bb9788606a844aeac11ef74e3475047 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:18:06.234984 29136 raft_consensus.cc:3037] T 5557c818bab749349ba412c7d6cd73d2 P d17b19f91f0c44a48c760e9cbf12de0a [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:18:06.235720 28859 raft_consensus.cc:2454] T 5557c818bab749349ba412c7d6cd73d2 P 8bb9788606a844aeac11ef74e3475047 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8ef706c3c2141aa895ff4a4b0b3a3da in term 1.
I20260929 03:18:06.235754 29136 raft_consensus.cc:2454] T 5557c818bab749349ba412c7d6cd73d2 P d17b19f91f0c44a48c760e9cbf12de0a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8ef706c3c2141aa895ff4a4b0b3a3da in term 1.
I20260929 03:18:06.235898 28932 leader_election.cc:302] T 5557c818bab749349ba412c7d6cd73d2 P c8ef706c3c2141aa895ff4a4b0b3a3da [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c8ef706c3c2141aa895ff4a4b0b3a3da, d17b19f91f0c44a48c760e9cbf12de0a; no voters:
I20260929 03:18:06.235989 29218 raft_consensus.cc:2781] T 5557c818bab749349ba412c7d6cd73d2 P c8ef706c3c2141aa895ff4a4b0b3a3da [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:18:06.236102 29218 raft_consensus.cc:686] T 5557c818bab749349ba412c7d6cd73d2 P c8ef706c3c2141aa895ff4a4b0b3a3da [term 1 LEADER]: Becoming Leader. State: Replica: c8ef706c3c2141aa895ff4a4b0b3a3da, State: Running, Role: LEADER
I20260929 03:18:06.236187 29218 consensus_queue.cc:237] T 5557c818bab749349ba412c7d6cd73d2 P c8ef706c3c2141aa895ff4a4b0b3a3da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bb9788606a844aeac11ef74e3475047" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 46097 } } peers { permanent_uuid: "c8ef706c3c2141aa895ff4a4b0b3a3da" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 44909 } } peers { permanent_uuid: "d17b19f91f0c44a48c760e9cbf12de0a" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 45077 } }
I20260929 03:18:06.236876 28703 catalog_manager.cc:5495] T 5557c818bab749349ba412c7d6cd73d2 P c8ef706c3c2141aa895ff4a4b0b3a3da reported cstate change: term changed from 0 to 1, leader changed from <none> to c8ef706c3c2141aa895ff4a4b0b3a3da (127.20.178.66). New cstate: current_term: 1 leader_uuid: "c8ef706c3c2141aa895ff4a4b0b3a3da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bb9788606a844aeac11ef74e3475047" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 46097 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8ef706c3c2141aa895ff4a4b0b3a3da" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 44909 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d17b19f91f0c44a48c760e9cbf12de0a" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 45077 } health_report { overall_health: UNKNOWN } } }
Sep 29 03:18:06 dist-test-slave-c9qh krb5kdc[28659](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651885, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.178.126@KRBTEST.COM for oryx/127.20.178.126@KRBTEST.COM
I20260929 03:18:06.312582 29249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:06.305422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:42107 (local address 127.0.0.1:57102)
0929 03:18:06.305571 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:06.305580 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 03:18:06.305625 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:18:06.305769 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:18:06.305770 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:18:06.306073 (+ 303us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:18:06.306119 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:06.306121 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:06.306810 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:06.306811 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:06.307315 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:06.307316 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:06.307712 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:06.307713 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:06.307751 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:06.308237 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:18:06.308242 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:18:06.309854 (+ 1612us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:18:06.311615 (+ 1761us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:18:06.311621 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:18:06.311623 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:18:06.311839 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:18:06.312017 (+ 178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:18:06.312018 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:18:06.312018 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:18:06.312068 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:18:06.312254 (+ 186us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:18:06.312258 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:18:06.312294 (+ 36us) client_negotiation.cc:769] Sending connection context
0929 03:18:06.312341 (+ 47us) client_negotiation.cc:240] Negotiation successful
0929 03:18:06.312468 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":67,"threads_started":1}
I20260929 03:18:06.312709 29250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:06.305514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57102 (local address 127.20.178.126:42107)
0929 03:18:06.305636 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0929 03:18:06.305638 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:18:06.305645 (+ 7us) server_negotiation.cc:407] Connection header received
0929 03:18:06.305667 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:18:06.305667 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:18:06.305676 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:18:06.305747 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:18:06.306206 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:06.306743 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:06.307398 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:06.307583 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:06.307677 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:06.309938 (+ 2261us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:18:06.309940 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:18:06.309940 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:18:06.309948 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:18:06.311544 (+ 1596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:18:06.311906 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:18:06.311906 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:18:06.311907 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:18:06.311961 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:18:06.312118 (+ 157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:18:06.312119 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:18:06.312119 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:18:06.312215 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:18:06.312263 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0929 03:18:06.312425 (+ 162us) server_negotiation.cc:299] Negotiation successful
0929 03:18:06.312518 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":25,"server-negotiator.queue_time_us":104,"thread_start_us":72,"threads_started":1}
I20260929 03:18:06.314536 28703 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.20.178.126@KRBTEST.COM'} at 127.0.0.1:57102:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260929 03:18:06.314760 28703 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 03:18:06.316148 29121 tablet_service.cc:1467] Processing CreateTablet for tablet 4b34e6c0493f4e9e8e73fee51997b0a3 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ae71d5a573fa402db692ac1dc11304c9]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 03:18:06.316141 28981 tablet_service.cc:1467] Processing CreateTablet for tablet 4b34e6c0493f4e9e8e73fee51997b0a3 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ae71d5a573fa402db692ac1dc11304c9]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 03:18:06.316166 28841 tablet_service.cc:1467] Processing CreateTablet for tablet 4b34e6c0493f4e9e8e73fee51997b0a3 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ae71d5a573fa402db692ac1dc11304c9]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 03:18:06.316301 28981 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b34e6c0493f4e9e8e73fee51997b0a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:18:06.316303 29121 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b34e6c0493f4e9e8e73fee51997b0a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:18:06.316315 28841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b34e6c0493f4e9e8e73fee51997b0a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:18:06.317509 29216 tablet_bootstrap.cc:492] T 4b34e6c0493f4e9e8e73fee51997b0a3 P 8bb9788606a844aeac11ef74e3475047: Bootstrap starting.
I20260929 03:18:06.317754 29212 tablet_bootstrap.cc:492] T 4b34e6c0493f4e9e8e73fee51997b0a3 P d17b19f91f0c44a48c760e9cbf12de0a: Bootstrap starting.
I20260929 03:18:06.317781 29213 tablet_bootstrap.cc:492] T 4b34e6c0493f4e9e8e73fee51997b0a3 P c8ef706c3c2141aa895ff4a4b0b3a3da: Bootstrap starting.
I20260929 03:18:06.318007 29216 tablet_bootstrap.cc:654] T 4b34e6c0493f4e9e8e73fee51997b0a3 P 8bb9788606a844aeac11ef74e3475047: Neither blocks nor log segments found. Creating new log.
I20260929 03:18:06.318041 29212 tablet_bootstrap.cc:654] T 4b34e6c0493f4e9e8e73fee51997b0a3 P d17b19f91f0c44a48c760e9cbf12de0a: Neither blocks nor log segments found. Creating new log.
I20260929 03:18:06.318204 29213 tablet_bootstrap.cc:654] T 4b34e6c0493f4e9e8e73fee51997b0a3 P c8ef706c3c2141aa895ff4a4b0b3a3da: Neither blocks nor log segments found. Creating new log.
I20260929 03:18:06.318549 29212 tablet_bootstrap.cc:492] T 4b34e6c0493f4e9e8e73fee51997b0a3 P d17b19f91f0c44a48c760e9cbf12de0a: No bootstrap required, opened a new log
I20260929 03:18:06.318590 29212 ts_tablet_manager.cc:1397] T 4b34e6c0493f4e9e8e73fee51997b0a3 P d17b19f91f0c44a48c760e9cbf12de0a: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260929 03:18:06.318706 29212 raft_consensus.cc:348] T 4b34e6c0493f4e9e8e73fee51997b0a3 P d17b19f91f0c44a48c760e9cbf12de0a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8ef706c3c2141aa895ff4a4b0b3a3da" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 44909 } } peers { permanent_uuid: "8bb9788606a844aeac11ef74e3475047" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 46097 } } peers { permanent_uuid: "d17b19f91f0c44a48c760e9cbf12de0a" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 45077 } }
I20260929 03:18:06.318763 29212 raft_consensus.cc:374] T 4b34e6c0493f4e9e8e73fee51997b0a3 P d17b19f91f0c44a48c760e9cbf12de0a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:18:06.318778 29212 raft_consensus.cc:729] T 4b34e6c0493f4e9e8e73fee51997b0a3 P d17b19f91f0c44a48c760e9cbf12de0a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d17b19f91f0c44a48c760e9cbf12de0a, State: Initialized, Role: FOLLOWER
I20260929 03:18:06.318838 29213 tablet_bootstrap.cc:492] T 4b34e6c0493f4e9e8e73fee51997b0a3 P c8ef706c3c2141aa895ff4a4b0b3a3da: No bootstrap required, opened a new log
I20260929 03:18:06.318843 29212 consensus_queue.cc:260] T 4b34e6c0493f4e9e8e73fee51997b0a3 P d17b19f91f0c44a48c760e9cbf12de0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8ef706c3c2141aa895ff4a4b0b3a3da" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 44909 } } peers { permanent_uuid: "8bb9788606a844aeac11ef74e3475047" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 46097 } } peers { permanent_uuid: "d17b19f91f0c44a48c760e9cbf12de0a" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 45077 } }
I20260929 03:18:06.318892 29213 ts_tablet_manager.cc:1397] T 4b34e6c0493f4e9e8e73fee51997b0a3 P c8ef706c3c2141aa895ff4a4b0b3a3da: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260929 03:18:06.318934 29212 ts_tablet_manager.cc:1428] T 4b34e6c0493f4e9e8e73fee51997b0a3 P d17b19f91f0c44a48c760e9cbf12de0a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260929 03:18:06.318962 29222 tablet_replica.cc:439] T 4b34e6c0493f4e9e8e73fee51997b0a3 P d17b19f91f0c44a48c760e9cbf12de0a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8ef706c3c2141aa895ff4a4b0b3a3da" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 44909 } } peers { permanent_uuid: "8bb9788606a844aeac11ef74e3475047" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 46097 } } peers { permanent_uuid: "d17b19f91f0c44a48c760e9cbf12de0a" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 45077 } } }
I20260929 03:18:06.319060 29222 tablet_replica.cc:442] T 4b34e6c0493f4e9e8e73fee51997b0a3 P d17b19f91f0c44a48c760e9cbf12de0a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 03:18:06.319038 29213 raft_consensus.cc:348] T 4b34e6c0493f4e9e8e73fee51997b0a3 P c8ef706c3c2141aa895ff4a4b0b3a3da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8ef706c3c2141aa895ff4a4b0b3a3da" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 44909 } } peers { permanent_uuid: "8bb9788606a844aeac11ef74e3475047" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 46097 } } peers { permanent_uuid: "d17b19f91f0c44a48c760e9cbf12de0a" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 45077 } }
I20260929 03:18:06.319099 29213 raft_consensus.cc:374] T 4b34e6c0493f4e9e8e73fee51997b0a3 P c8ef706c3c2141aa895ff4a4b0b3a3da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:18:06.319135 29213 raft_consensus.cc:729] T 4b34e6c0493f4e9e8e73fee51997b0a3 P c8ef706c3c2141aa895ff4a4b0b3a3da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8ef706c3c2141aa895ff4a4b0b3a3da, State: Initialized, Role: FOLLOWER
I20260929 03:18:06.319177 29213 consensus_queue.cc:260] T 4b34e6c0493f4e9e8e73fee51997b0a3 P c8ef706c3c2141aa895ff4a4b0b3a3da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8ef706c3c2141aa895ff4a4b0b3a3da" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 44909 } } peers { permanent_uuid: "8bb9788606a844aeac11ef74e3475047" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 46097 } } peers { permanent_uuid: "d17b19f91f0c44a48c760e9cbf12de0a" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 45077 } }
I20260929 03:18:06.319284 29213 ts_tablet_manager.cc:1428] T 4b34e6c0493f4e9e8e73fee51997b0a3 P c8ef706c3c2141aa895ff4a4b0b3a3da: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260929 03:18:06.319396 29216 tablet_bootstrap.cc:492] T 4b34e6c0493f4e9e8e73fee51997b0a3 P 8bb9788606a844aeac11ef74e3475047: No bootstrap required, opened a new log
I20260929 03:18:06.319442 29216 ts_tablet_manager.cc:1397] T 4b34e6c0493f4e9e8e73fee51997b0a3 P 8bb9788606a844aeac11ef74e3475047: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260929 03:18:06.319423 29218 tablet_replica.cc:439] T 4b34e6c0493f4e9e8e73fee51997b0a3 P c8ef706c3c2141aa895ff4a4b0b3a3da: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8ef706c3c2141aa895ff4a4b0b3a3da" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 44909 } } peers { permanent_uuid: "8bb9788606a844aeac11ef74e3475047" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 46097 } } peers { permanent_uuid: "d17b19f91f0c44a48c760e9cbf12de0a" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 45077 } } }
I20260929 03:18:06.319494 29218 tablet_replica.cc:442] T 4b34e6c0493f4e9e8e73fee51997b0a3 P c8ef706c3c2141aa895ff4a4b0b3a3da: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 03:18:06.319577 29216 raft_consensus.cc:348] T 4b34e6c0493f4e9e8e73fee51997b0a3 P 8bb9788606a844aeac11ef74e3475047 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8ef706c3c2141aa895ff4a4b0b3a3da" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 44909 } } peers { permanent_uuid: "8bb9788606a844aeac11ef74e3475047" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 46097 } } peers { permanent_uuid: "d17b19f91f0c44a48c760e9cbf12de0a" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 45077 } }
I20260929 03:18:06.319674 29216 raft_consensus.cc:374] T 4b34e6c0493f4e9e8e73fee51997b0a3 P 8bb9788606a844aeac11ef74e3475047 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:18:06.319701 29216 raft_consensus.cc:729] T 4b34e6c0493f4e9e8e73fee51997b0a3 P 8bb9788606a844aeac11ef74e3475047 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8bb9788606a844aeac11ef74e3475047, State: Initialized, Role: FOLLOWER
I20260929 03:18:06.319757 29216 consensus_queue.cc:260] T 4b34e6c0493f4e9e8e73fee51997b0a3 P 8bb9788606a844aeac11ef74e3475047 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8ef706c3c2141aa895ff4a4b0b3a3da" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 44909 } } peers { permanent_uuid: "8bb9788606a844aeac11ef74e3475047" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 46097 } } peers { permanent_uuid: "d17b19f91f0c44a48c760e9cbf12de0a" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 45077 } }
I20260929 03:18:06.319890 29216 ts_tablet_manager.cc:1428] T 4b34e6c0493f4e9e8e73fee51997b0a3 P 8bb9788606a844aeac11ef74e3475047: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260929 03:18:06.319954 29220 tablet_replica.cc:439] T 4b34e6c0493f4e9e8e73fee51997b0a3 P 8bb9788606a844aeac11ef74e3475047: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8ef706c3c2141aa895ff4a4b0b3a3da" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 44909 } } peers { permanent_uuid: "8bb9788606a844aeac11ef74e3475047" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 46097 } } peers { permanent_uuid: "d17b19f91f0c44a48c760e9cbf12de0a" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 45077 } } }
I20260929 03:18:06.320030 29220 tablet_replica.cc:442] T 4b34e6c0493f4e9e8e73fee51997b0a3 P 8bb9788606a844aeac11ef74e3475047: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 03:18:06.476851 29218 raft_consensus.cc:482] T 4b34e6c0493f4e9e8e73fee51997b0a3 P c8ef706c3c2141aa895ff4a4b0b3a3da [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:18:06.476965 29218 raft_consensus.cc:504] T 4b34e6c0493f4e9e8e73fee51997b0a3 P c8ef706c3c2141aa895ff4a4b0b3a3da [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8ef706c3c2141aa895ff4a4b0b3a3da" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 44909 } } peers { permanent_uuid: "8bb9788606a844aeac11ef74e3475047" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 46097 } } peers { permanent_uuid: "d17b19f91f0c44a48c760e9cbf12de0a" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 45077 } }
I20260929 03:18:06.477149 29218 leader_election.cc:288] T 4b34e6c0493f4e9e8e73fee51997b0a3 P c8ef706c3c2141aa895ff4a4b0b3a3da [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8bb9788606a844aeac11ef74e3475047 (127.20.178.65:46097), d17b19f91f0c44a48c760e9cbf12de0a (127.20.178.67:45077)
I20260929 03:18:06.477442 29136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b34e6c0493f4e9e8e73fee51997b0a3" candidate_uuid: "c8ef706c3c2141aa895ff4a4b0b3a3da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d17b19f91f0c44a48c760e9cbf12de0a" is_pre_election: true
I20260929 03:18:06.477442 28859 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b34e6c0493f4e9e8e73fee51997b0a3" candidate_uuid: "c8ef706c3c2141aa895ff4a4b0b3a3da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8bb9788606a844aeac11ef74e3475047" is_pre_election: true
I20260929 03:18:06.477543 28859 raft_consensus.cc:2454] T 4b34e6c0493f4e9e8e73fee51997b0a3 P 8bb9788606a844aeac11ef74e3475047 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8ef706c3c2141aa895ff4a4b0b3a3da in term 0.
I20260929 03:18:06.477543 29136 raft_consensus.cc:2454] T 4b34e6c0493f4e9e8e73fee51997b0a3 P d17b19f91f0c44a48c760e9cbf12de0a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8ef706c3c2141aa895ff4a4b0b3a3da in term 0.
I20260929 03:18:06.477722 28932 leader_election.cc:302] T 4b34e6c0493f4e9e8e73fee51997b0a3 P c8ef706c3c2141aa895ff4a4b0b3a3da [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c8ef706c3c2141aa895ff4a4b0b3a3da, d17b19f91f0c44a48c760e9cbf12de0a; no voters:
I20260929 03:18:06.477815 29218 raft_consensus.cc:2781] T 4b34e6c0493f4e9e8e73fee51997b0a3 P c8ef706c3c2141aa895ff4a4b0b3a3da [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 03:18:06.477874 29218 raft_consensus.cc:482] T 4b34e6c0493f4e9e8e73fee51997b0a3 P c8ef706c3c2141aa895ff4a4b0b3a3da [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 03:18:06.477905 29218 raft_consensus.cc:3037] T 4b34e6c0493f4e9e8e73fee51997b0a3 P c8ef706c3c2141aa895ff4a4b0b3a3da [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:18:06.478812 29218 raft_consensus.cc:504] T 4b34e6c0493f4e9e8e73fee51997b0a3 P c8ef706c3c2141aa895ff4a4b0b3a3da [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8ef706c3c2141aa895ff4a4b0b3a3da" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 44909 } } peers { permanent_uuid: "8bb9788606a844aeac11ef74e3475047" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 46097 } } peers { permanent_uuid: "d17b19f91f0c44a48c760e9cbf12de0a" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 45077 } }
I20260929 03:18:06.478945 29218 leader_election.cc:288] T 4b34e6c0493f4e9e8e73fee51997b0a3 P c8ef706c3c2141aa895ff4a4b0b3a3da [CANDIDATE]: Term 1 election: Requested vote from peers 8bb9788606a844aeac11ef74e3475047 (127.20.178.65:46097), d17b19f91f0c44a48c760e9cbf12de0a (127.20.178.67:45077)
I20260929 03:18:06.479108 28859 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b34e6c0493f4e9e8e73fee51997b0a3" candidate_uuid: "c8ef706c3c2141aa895ff4a4b0b3a3da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8bb9788606a844aeac11ef74e3475047"
I20260929 03:18:06.479116 29136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b34e6c0493f4e9e8e73fee51997b0a3" candidate_uuid: "c8ef706c3c2141aa895ff4a4b0b3a3da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d17b19f91f0c44a48c760e9cbf12de0a"
I20260929 03:18:06.479182 29136 raft_consensus.cc:3037] T 4b34e6c0493f4e9e8e73fee51997b0a3 P d17b19f91f0c44a48c760e9cbf12de0a [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:18:06.479184 28859 raft_consensus.cc:3037] T 4b34e6c0493f4e9e8e73fee51997b0a3 P 8bb9788606a844aeac11ef74e3475047 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:18:06.479782 29136 raft_consensus.cc:2454] T 4b34e6c0493f4e9e8e73fee51997b0a3 P d17b19f91f0c44a48c760e9cbf12de0a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8ef706c3c2141aa895ff4a4b0b3a3da in term 1.
I20260929 03:18:06.479857 28859 raft_consensus.cc:2454] T 4b34e6c0493f4e9e8e73fee51997b0a3 P 8bb9788606a844aeac11ef74e3475047 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8ef706c3c2141aa895ff4a4b0b3a3da in term 1.
I20260929 03:18:06.479938 28932 leader_election.cc:302] T 4b34e6c0493f4e9e8e73fee51997b0a3 P c8ef706c3c2141aa895ff4a4b0b3a3da [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c8ef706c3c2141aa895ff4a4b0b3a3da, d17b19f91f0c44a48c760e9cbf12de0a; no voters:
I20260929 03:18:06.480033 29218 raft_consensus.cc:2781] T 4b34e6c0493f4e9e8e73fee51997b0a3 P c8ef706c3c2141aa895ff4a4b0b3a3da [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:18:06.480087 29218 raft_consensus.cc:686] T 4b34e6c0493f4e9e8e73fee51997b0a3 P c8ef706c3c2141aa895ff4a4b0b3a3da [term 1 LEADER]: Becoming Leader. State: Replica: c8ef706c3c2141aa895ff4a4b0b3a3da, State: Running, Role: LEADER
I20260929 03:18:06.480129 29218 consensus_queue.cc:237] T 4b34e6c0493f4e9e8e73fee51997b0a3 P c8ef706c3c2141aa895ff4a4b0b3a3da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8ef706c3c2141aa895ff4a4b0b3a3da" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 44909 } } peers { permanent_uuid: "8bb9788606a844aeac11ef74e3475047" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 46097 } } peers { permanent_uuid: "d17b19f91f0c44a48c760e9cbf12de0a" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 45077 } }
I20260929 03:18:06.480417 29242 tablet_replica.cc:439] T 4b34e6c0493f4e9e8e73fee51997b0a3 P c8ef706c3c2141aa895ff4a4b0b3a3da: TxnStatusTablet state changed. Reason: New leader c8ef706c3c2141aa895ff4a4b0b3a3da. Latest consensus state: current_term: 1 leader_uuid: "c8ef706c3c2141aa895ff4a4b0b3a3da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8ef706c3c2141aa895ff4a4b0b3a3da" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 44909 } } peers { permanent_uuid: "8bb9788606a844aeac11ef74e3475047" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 46097 } } peers { permanent_uuid: "d17b19f91f0c44a48c760e9cbf12de0a" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 45077 } } }
I20260929 03:18:06.480508 29242 tablet_replica.cc:442] T 4b34e6c0493f4e9e8e73fee51997b0a3 P c8ef706c3c2141aa895ff4a4b0b3a3da: This TxnStatusTablet replica's current role is: LEADER
I20260929 03:18:06.480810 28704 catalog_manager.cc:5495] T 4b34e6c0493f4e9e8e73fee51997b0a3 P c8ef706c3c2141aa895ff4a4b0b3a3da reported cstate change: term changed from 0 to 1, leader changed from <none> to c8ef706c3c2141aa895ff4a4b0b3a3da (127.20.178.66). New cstate: current_term: 1 leader_uuid: "c8ef706c3c2141aa895ff4a4b0b3a3da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8ef706c3c2141aa895ff4a4b0b3a3da" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 44909 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8bb9788606a844aeac11ef74e3475047" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 46097 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d17b19f91f0c44a48c760e9cbf12de0a" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 45077 } health_report { overall_health: UNKNOWN } } }
I20260929 03:18:06.481115 29255 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260929 03:18:06.481252 29255 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260929 03:18:06.503000 29258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:06.500133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33516 (local address 127.20.178.66:44909)
0929 03:18:06.500292 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0929 03:18:06.500294 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:18:06.500302 (+ 8us) server_negotiation.cc:407] Connection header received
0929 03:18:06.500338 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:18:06.500339 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:18:06.500360 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:18:06.500440 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:18:06.500570 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:06.501121 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:06.502155 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:06.502379 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:06.502461 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:06.502649 (+ 188us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:18:06.502752 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:18:06.502833 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0929 03:18:06.502876 (+ 43us) server_negotiation.cc:299] Negotiation successful
0929 03:18:06.502898 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":72,"threads_started":1}
I20260929 03:18:06.502992 29249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:06.500057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.66:44909 (local address 127.0.0.1:33516)
0929 03:18:06.500130 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:06.500141 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 03:18:06.500192 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:18:06.500445 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:18:06.500447 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:18:06.500448 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 03:18:06.500497 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:06.500500 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:06.501211 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:06.501212 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:06.502063 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:06.502065 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:06.502462 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:06.502463 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:06.502520 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:06.502536 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 03:18:06.502818 (+ 282us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 03:18:06.502826 (+ 8us) client_negotiation.cc:769] Sending connection context
0929 03:18:06.502875 (+ 49us) client_negotiation.cc:240] Negotiation successful
0929 03:18:06.502921 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260929 03:18:06.504199 28859 raft_consensus.cc:1261] T 4b34e6c0493f4e9e8e73fee51997b0a3 P 8bb9788606a844aeac11ef74e3475047 [term 1 FOLLOWER]: Refusing update from remote peer c8ef706c3c2141aa895ff4a4b0b3a3da: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:18:06.504253 29136 raft_consensus.cc:1261] T 4b34e6c0493f4e9e8e73fee51997b0a3 P d17b19f91f0c44a48c760e9cbf12de0a [term 1 FOLLOWER]: Refusing update from remote peer c8ef706c3c2141aa895ff4a4b0b3a3da: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:18:06.504567 29218 consensus_queue.cc:1035] T 4b34e6c0493f4e9e8e73fee51997b0a3 P c8ef706c3c2141aa895ff4a4b0b3a3da [LEADER]: Connected to new peer: Peer: permanent_uuid: "8bb9788606a844aeac11ef74e3475047" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 46097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:18:06.504709 29242 consensus_queue.cc:1035] T 4b34e6c0493f4e9e8e73fee51997b0a3 P c8ef706c3c2141aa895ff4a4b0b3a3da [LEADER]: Connected to new peer: Peer: permanent_uuid: "d17b19f91f0c44a48c760e9cbf12de0a" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 45077 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:18:06.505388 29222 tablet_replica.cc:439] T 4b34e6c0493f4e9e8e73fee51997b0a3 P d17b19f91f0c44a48c760e9cbf12de0a: TxnStatusTablet state changed. Reason: New leader c8ef706c3c2141aa895ff4a4b0b3a3da. Latest consensus state: current_term: 1 leader_uuid: "c8ef706c3c2141aa895ff4a4b0b3a3da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8ef706c3c2141aa895ff4a4b0b3a3da" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 44909 } } peers { permanent_uuid: "8bb9788606a844aeac11ef74e3475047" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 46097 } } peers { permanent_uuid: "d17b19f91f0c44a48c760e9cbf12de0a" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 45077 } } }
I20260929 03:18:06.505481 29222 tablet_replica.cc:442] T 4b34e6c0493f4e9e8e73fee51997b0a3 P d17b19f91f0c44a48c760e9cbf12de0a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 03:18:06.505875 29220 tablet_replica.cc:439] T 4b34e6c0493f4e9e8e73fee51997b0a3 P 8bb9788606a844aeac11ef74e3475047: TxnStatusTablet state changed. Reason: New leader c8ef706c3c2141aa895ff4a4b0b3a3da. Latest consensus state: current_term: 1 leader_uuid: "c8ef706c3c2141aa895ff4a4b0b3a3da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8ef706c3c2141aa895ff4a4b0b3a3da" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 44909 } } peers { permanent_uuid: "8bb9788606a844aeac11ef74e3475047" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 46097 } } peers { permanent_uuid: "d17b19f91f0c44a48c760e9cbf12de0a" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 45077 } } }
I20260929 03:18:06.505964 29220 tablet_replica.cc:442] T 4b34e6c0493f4e9e8e73fee51997b0a3 P 8bb9788606a844aeac11ef74e3475047: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 03:18:06.506454 29218 tablet_replica.cc:439] T 4b34e6c0493f4e9e8e73fee51997b0a3 P c8ef706c3c2141aa895ff4a4b0b3a3da: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c8ef706c3c2141aa895ff4a4b0b3a3da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8ef706c3c2141aa895ff4a4b0b3a3da" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 44909 } } peers { permanent_uuid: "8bb9788606a844aeac11ef74e3475047" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 46097 } } peers { permanent_uuid: "d17b19f91f0c44a48c760e9cbf12de0a" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 45077 } } }
I20260929 03:18:06.506536 29218 tablet_replica.cc:442] T 4b34e6c0493f4e9e8e73fee51997b0a3 P c8ef706c3c2141aa895ff4a4b0b3a3da: This TxnStatusTablet replica's current role is: LEADER
I20260929 03:18:06.506623 29222 tablet_replica.cc:439] T 4b34e6c0493f4e9e8e73fee51997b0a3 P d17b19f91f0c44a48c760e9cbf12de0a: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c8ef706c3c2141aa895ff4a4b0b3a3da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8ef706c3c2141aa895ff4a4b0b3a3da" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 44909 } } peers { permanent_uuid: "8bb9788606a844aeac11ef74e3475047" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 46097 } } peers { permanent_uuid: "d17b19f91f0c44a48c760e9cbf12de0a" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 45077 } } }
I20260929 03:18:06.506698 29222 tablet_replica.cc:442] T 4b34e6c0493f4e9e8e73fee51997b0a3 P d17b19f91f0c44a48c760e9cbf12de0a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 03:18:06.507179 29242 tablet_replica.cc:439] T 4b34e6c0493f4e9e8e73fee51997b0a3 P c8ef706c3c2141aa895ff4a4b0b3a3da: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c8ef706c3c2141aa895ff4a4b0b3a3da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8ef706c3c2141aa895ff4a4b0b3a3da" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 44909 } } peers { permanent_uuid: "8bb9788606a844aeac11ef74e3475047" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 46097 } } peers { permanent_uuid: "d17b19f91f0c44a48c760e9cbf12de0a" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 45077 } } }
I20260929 03:18:06.507253 29242 tablet_replica.cc:442] T 4b34e6c0493f4e9e8e73fee51997b0a3 P c8ef706c3c2141aa895ff4a4b0b3a3da: This TxnStatusTablet replica's current role is: LEADER
I20260929 03:18:06.507655 29220 tablet_replica.cc:439] T 4b34e6c0493f4e9e8e73fee51997b0a3 P 8bb9788606a844aeac11ef74e3475047: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c8ef706c3c2141aa895ff4a4b0b3a3da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8ef706c3c2141aa895ff4a4b0b3a3da" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 44909 } } peers { permanent_uuid: "8bb9788606a844aeac11ef74e3475047" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 46097 } } peers { permanent_uuid: "d17b19f91f0c44a48c760e9cbf12de0a" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 45077 } } }
I20260929 03:18:06.507736 29220 tablet_replica.cc:442] T 4b34e6c0493f4e9e8e73fee51997b0a3 P 8bb9788606a844aeac11ef74e3475047: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 03:18:06.510970 29258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:06.508675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33528 (local address 127.20.178.66:44909)
0929 03:18:06.508719 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0929 03:18:06.508722 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:18:06.508817 (+ 95us) server_negotiation.cc:407] Connection header received
0929 03:18:06.508892 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:18:06.508893 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:18:06.508903 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:18:06.508956 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:18:06.509101 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:06.509553 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:06.510286 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:06.510486 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:06.510555 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:06.510720 (+ 165us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:18:06.510750 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:18:06.510817 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0929 03:18:06.510877 (+ 60us) server_negotiation.cc:299] Negotiation successful
0929 03:18:06.510895 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260929 03:18:06.517486 29258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:06.514912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.66:47093 (local address 127.20.178.66:44909)
0929 03:18:06.514954 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0929 03:18:06.514956 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:18:06.514986 (+ 30us) server_negotiation.cc:407] Connection header received
0929 03:18:06.515072 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:18:06.515072 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:18:06.515082 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:18:06.515150 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:18:06.515272 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:06.515974 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:06.516738 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:06.517036 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:06.517101 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:06.517263 (+ 162us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:18:06.517292 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:18:06.517346 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0929 03:18:06.517389 (+ 43us) server_negotiation.cc:299] Negotiation successful
0929 03:18:06.517408 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260929 03:18:06.517535 29271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:06.514792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.66:44909 (local address 127.20.178.66:47093)
0929 03:18:06.514933 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:06.514942 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 03:18:06.515001 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:18:06.515156 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:18:06.515157 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:18:06.515158 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 03:18:06.515204 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:06.515205 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:06.516043 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:06.516043 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:06.516571 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:06.516573 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:06.517130 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:06.517131 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:06.517176 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:06.517189 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 03:18:06.517351 (+ 162us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 03:18:06.517352 (+ 1us) client_negotiation.cc:769] Sending connection context
0929 03:18:06.517386 (+ 34us) client_negotiation.cc:240] Negotiation successful
0929 03:18:06.517418 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":84,"threads_started":1}
I20260929 03:18:06.519536 29136 raft_consensus.cc:1261] T 5557c818bab749349ba412c7d6cd73d2 P d17b19f91f0c44a48c760e9cbf12de0a [term 1 FOLLOWER]: Refusing update from remote peer c8ef706c3c2141aa895ff4a4b0b3a3da: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:18:06.519546 28859 raft_consensus.cc:1261] T 5557c818bab749349ba412c7d6cd73d2 P 8bb9788606a844aeac11ef74e3475047 [term 1 FOLLOWER]: Refusing update from remote peer c8ef706c3c2141aa895ff4a4b0b3a3da: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:18:06.519732 29242 consensus_queue.cc:1035] T 5557c818bab749349ba412c7d6cd73d2 P c8ef706c3c2141aa895ff4a4b0b3a3da [LEADER]: Connected to new peer: Peer: permanent_uuid: "d17b19f91f0c44a48c760e9cbf12de0a" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 45077 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:18:06.519853 29272 consensus_queue.cc:1035] T 5557c818bab749349ba412c7d6cd73d2 P c8ef706c3c2141aa895ff4a4b0b3a3da [LEADER]: Connected to new peer: Peer: permanent_uuid: "8bb9788606a844aeac11ef74e3475047" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 46097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:18:06.531047 28703 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57098:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:18:06.531126 28703 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57098:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:18:06.531867 28703 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b8fefe3633e6491689c51b85ebdc7336: Sending DeleteTablet for 3 replicas of tablet 5557c818bab749349ba412c7d6cd73d2
I20260929 03:18:06.532155 28981 tablet_service.cc:1514] Processing DeleteTablet for tablet 5557c818bab749349ba412c7d6cd73d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:18:06 UTC) from {username='oryx', principal='oryx/127.20.178.126@KRBTEST.COM'} at 127.0.0.1:33508
I20260929 03:18:06.532147 29121 tablet_service.cc:1514] Processing DeleteTablet for tablet 5557c818bab749349ba412c7d6cd73d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:18:06 UTC) from {username='oryx', principal='oryx/127.20.178.126@KRBTEST.COM'} at 127.0.0.1:38304
I20260929 03:18:06.532160 28841 tablet_service.cc:1514] Processing DeleteTablet for tablet 5557c818bab749349ba412c7d6cd73d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:18:06 UTC) from {username='oryx', principal='oryx/127.20.178.126@KRBTEST.COM'} at 127.0.0.1:47574
I20260929 03:18:06.532403 29277 tablet_replica.cc:330] T 5557c818bab749349ba412c7d6cd73d2 P 8bb9788606a844aeac11ef74e3475047: stopping tablet replica
I20260929 03:18:06.532346 29276 tablet_replica.cc:330] T 5557c818bab749349ba412c7d6cd73d2 P d17b19f91f0c44a48c760e9cbf12de0a: stopping tablet replica
I20260929 03:18:06.532507 29277 raft_consensus.cc:2229] T 5557c818bab749349ba412c7d6cd73d2 P 8bb9788606a844aeac11ef74e3475047 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:18:06.532723 29277 raft_consensus.cc:2258] T 5557c818bab749349ba412c7d6cd73d2 P 8bb9788606a844aeac11ef74e3475047 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:18:06.532992 29277 ts_tablet_manager.cc:1905] T 5557c818bab749349ba412c7d6cd73d2 P 8bb9788606a844aeac11ef74e3475047: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:18:06.532482 29276 raft_consensus.cc:2229] T 5557c818bab749349ba412c7d6cd73d2 P d17b19f91f0c44a48c760e9cbf12de0a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:18:06.534472 29276 raft_consensus.cc:2258] T 5557c818bab749349ba412c7d6cd73d2 P d17b19f91f0c44a48c760e9cbf12de0a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:18:06.534754 29278 tablet_replica.cc:330] T 5557c818bab749349ba412c7d6cd73d2 P c8ef706c3c2141aa895ff4a4b0b3a3da: stopping tablet replica
I20260929 03:18:06.534761 29276 ts_tablet_manager.cc:1905] T 5557c818bab749349ba412c7d6cd73d2 P d17b19f91f0c44a48c760e9cbf12de0a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:18:06.534861 29278 raft_consensus.cc:2229] T 5557c818bab749349ba412c7d6cd73d2 P c8ef706c3c2141aa895ff4a4b0b3a3da [term 1 LEADER]: Raft consensus shutting down.
I20260929 03:18:06.534981 29278 raft_consensus.cc:2258] T 5557c818bab749349ba412c7d6cd73d2 P c8ef706c3c2141aa895ff4a4b0b3a3da [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:18:06.535245 29278 ts_tablet_manager.cc:1905] T 5557c818bab749349ba412c7d6cd73d2 P c8ef706c3c2141aa895ff4a4b0b3a3da: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:18:06.536056 29277 ts_tablet_manager.cc:1918] T 5557c818bab749349ba412c7d6cd73d2 P 8bb9788606a844aeac11ef74e3475047: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 03:18:06.536199 29277 log.cc:1196] T 5557c818bab749349ba412c7d6cd73d2 P 8bb9788606a844aeac11ef74e3475047: Deleting WAL directory at /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651852476208-21193-0/minicluster-data/ts-0/wal/wals/5557c818bab749349ba412c7d6cd73d2
I20260929 03:18:06.536237 29276 ts_tablet_manager.cc:1918] T 5557c818bab749349ba412c7d6cd73d2 P d17b19f91f0c44a48c760e9cbf12de0a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 03:18:06.536291 29276 log.cc:1196] T 5557c818bab749349ba412c7d6cd73d2 P d17b19f91f0c44a48c760e9cbf12de0a: Deleting WAL directory at /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651852476208-21193-0/minicluster-data/ts-2/wal/wals/5557c818bab749349ba412c7d6cd73d2
I20260929 03:18:06.536454 29277 ts_tablet_manager.cc:1939] T 5557c818bab749349ba412c7d6cd73d2 P 8bb9788606a844aeac11ef74e3475047: Deleting consensus metadata
I20260929 03:18:06.536490 29276 ts_tablet_manager.cc:1939] T 5557c818bab749349ba412c7d6cd73d2 P d17b19f91f0c44a48c760e9cbf12de0a: Deleting consensus metadata
I20260929 03:18:06.536790 28688 catalog_manager.cc:4841] TS 8bb9788606a844aeac11ef74e3475047 (127.20.178.65:46097): tablet 5557c818bab749349ba412c7d6cd73d2 (table test-table [id=fe0b77c11a034e00a99aeb971460b18f]) successfully deleted
I20260929 03:18:06.536897 28688 catalog_manager.cc:4841] TS d17b19f91f0c44a48c760e9cbf12de0a (127.20.178.67:45077): tablet 5557c818bab749349ba412c7d6cd73d2 (table test-table [id=fe0b77c11a034e00a99aeb971460b18f]) successfully deleted
I20260929 03:18:06.537513 29278 ts_tablet_manager.cc:1918] T 5557c818bab749349ba412c7d6cd73d2 P c8ef706c3c2141aa895ff4a4b0b3a3da: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 03:18:06.537616 29278 log.cc:1196] T 5557c818bab749349ba412c7d6cd73d2 P c8ef706c3c2141aa895ff4a4b0b3a3da: Deleting WAL directory at /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651852476208-21193-0/minicluster-data/ts-1/wal/wals/5557c818bab749349ba412c7d6cd73d2
I20260929 03:18:06.537921 29278 ts_tablet_manager.cc:1939] T 5557c818bab749349ba412c7d6cd73d2 P c8ef706c3c2141aa895ff4a4b0b3a3da: Deleting consensus metadata
I20260929 03:18:06.538232 28689 catalog_manager.cc:4841] TS c8ef706c3c2141aa895ff4a4b0b3a3da (127.20.178.66:44909): tablet 5557c818bab749349ba412c7d6cd73d2 (table test-table [id=fe0b77c11a034e00a99aeb971460b18f]) successfully deleted
Sep 29 03:18:06 dist-test-slave-c9qh krb5kdc[28659](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651886, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 03:18:06 dist-test-slave-c9qh krb5kdc[28659](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.20.178.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:18:06 dist-test-slave-c9qh krb5kdc[28659](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.20.178.126@KRBTEST.COM, Server not found in Kerberos database
W20260929 03:18:06.556793 29250 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:18:06.551110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57118 (local address 127.20.178.126:42107)
0929 03:18:06.551187 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0929 03:18:06.551189 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:18:06.551290 (+ 101us) server_negotiation.cc:407] Connection header received
0929 03:18:06.551374 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:18:06.551375 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:18:06.551386 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:18:06.551457 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:18:06.552041 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:06.552556 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:06.553345 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:06.553560 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:06.553668 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:06.556719 (+ 3051us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57118: BlockingRecv error: recv got EOF from 127.0.0.1:57118 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Sep 29 03:18:06 dist-test-slave-c9qh krb5kdc[28659](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651886, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.20.178.126@KRBTEST.COM
I20260929 03:18:06.565035 29250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:06.557611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57134 (local address 127.20.178.126:42107)
0929 03:18:06.557678 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0929 03:18:06.557679 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 03:18:06.557689 (+ 10us) server_negotiation.cc:407] Connection header received
0929 03:18:06.557751 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:18:06.557752 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:18:06.557761 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:18:06.557830 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:18:06.558233 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:06.558979 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:06.559682 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:06.559954 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:06.560028 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:06.562129 (+ 2101us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:18:06.562130 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:18:06.562131 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:18:06.562138 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:18:06.563731 (+ 1593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:18:06.564135 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:18:06.564136 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:18:06.564136 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:18:06.564196 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:18:06.564411 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:18:06.564412 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:18:06.564413 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:18:06.564552 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:18:06.564686 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0929 03:18:06.564860 (+ 174us) server_negotiation.cc:299] Negotiation successful
0929 03:18:06.564940 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 03:18:06.566756 28703 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57134:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 03:18:06.566926 28703 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 03:18:06.568279 28841 tablet_service.cc:1467] Processing CreateTablet for tablet 46ed7d7f48644de581f100003efeed67 (DEFAULT_TABLE table=test-table [id=1f2cc505dfaf40f4b3ce463cf119a8d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:18:06.568288 28981 tablet_service.cc:1467] Processing CreateTablet for tablet 46ed7d7f48644de581f100003efeed67 (DEFAULT_TABLE table=test-table [id=1f2cc505dfaf40f4b3ce463cf119a8d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:18:06.568341 29121 tablet_service.cc:1467] Processing CreateTablet for tablet 46ed7d7f48644de581f100003efeed67 (DEFAULT_TABLE table=test-table [id=1f2cc505dfaf40f4b3ce463cf119a8d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:18:06.568413 28841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46ed7d7f48644de581f100003efeed67. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:18:06.568413 28981 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46ed7d7f48644de581f100003efeed67. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:18:06.568434 29121 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46ed7d7f48644de581f100003efeed67. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:18:06.569749 29212 tablet_bootstrap.cc:492] T 46ed7d7f48644de581f100003efeed67 P d17b19f91f0c44a48c760e9cbf12de0a: Bootstrap starting.
I20260929 03:18:06.569749 29213 tablet_bootstrap.cc:492] T 46ed7d7f48644de581f100003efeed67 P c8ef706c3c2141aa895ff4a4b0b3a3da: Bootstrap starting.
I20260929 03:18:06.569844 29216 tablet_bootstrap.cc:492] T 46ed7d7f48644de581f100003efeed67 P 8bb9788606a844aeac11ef74e3475047: Bootstrap starting.
I20260929 03:18:06.570017 29212 tablet_bootstrap.cc:654] T 46ed7d7f48644de581f100003efeed67 P d17b19f91f0c44a48c760e9cbf12de0a: Neither blocks nor log segments found. Creating new log.
I20260929 03:18:06.570108 29213 tablet_bootstrap.cc:654] T 46ed7d7f48644de581f100003efeed67 P c8ef706c3c2141aa895ff4a4b0b3a3da: Neither blocks nor log segments found. Creating new log.
I20260929 03:18:06.570205 29216 tablet_bootstrap.cc:654] T 46ed7d7f48644de581f100003efeed67 P 8bb9788606a844aeac11ef74e3475047: Neither blocks nor log segments found. Creating new log.
I20260929 03:18:06.570621 29212 tablet_bootstrap.cc:492] T 46ed7d7f48644de581f100003efeed67 P d17b19f91f0c44a48c760e9cbf12de0a: No bootstrap required, opened a new log
I20260929 03:18:06.570706 29212 ts_tablet_manager.cc:1397] T 46ed7d7f48644de581f100003efeed67 P d17b19f91f0c44a48c760e9cbf12de0a: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260929 03:18:06.570854 29212 raft_consensus.cc:348] T 46ed7d7f48644de581f100003efeed67 P d17b19f91f0c44a48c760e9cbf12de0a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bb9788606a844aeac11ef74e3475047" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 46097 } } peers { permanent_uuid: "c8ef706c3c2141aa895ff4a4b0b3a3da" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 44909 } } peers { permanent_uuid: "d17b19f91f0c44a48c760e9cbf12de0a" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 45077 } }
I20260929 03:18:06.570927 29212 raft_consensus.cc:374] T 46ed7d7f48644de581f100003efeed67 P d17b19f91f0c44a48c760e9cbf12de0a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:18:06.570972 29212 raft_consensus.cc:729] T 46ed7d7f48644de581f100003efeed67 P d17b19f91f0c44a48c760e9cbf12de0a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d17b19f91f0c44a48c760e9cbf12de0a, State: Initialized, Role: FOLLOWER
I20260929 03:18:06.570966 29213 tablet_bootstrap.cc:492] T 46ed7d7f48644de581f100003efeed67 P c8ef706c3c2141aa895ff4a4b0b3a3da: No bootstrap required, opened a new log
I20260929 03:18:06.571009 29213 ts_tablet_manager.cc:1397] T 46ed7d7f48644de581f100003efeed67 P c8ef706c3c2141aa895ff4a4b0b3a3da: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260929 03:18:06.571014 29212 consensus_queue.cc:260] T 46ed7d7f48644de581f100003efeed67 P d17b19f91f0c44a48c760e9cbf12de0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bb9788606a844aeac11ef74e3475047" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 46097 } } peers { permanent_uuid: "c8ef706c3c2141aa895ff4a4b0b3a3da" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 44909 } } peers { permanent_uuid: "d17b19f91f0c44a48c760e9cbf12de0a" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 45077 } }
I20260929 03:18:06.571098 29212 ts_tablet_manager.cc:1428] T 46ed7d7f48644de581f100003efeed67 P d17b19f91f0c44a48c760e9cbf12de0a: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260929 03:18:06.571162 29213 raft_consensus.cc:348] T 46ed7d7f48644de581f100003efeed67 P c8ef706c3c2141aa895ff4a4b0b3a3da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bb9788606a844aeac11ef74e3475047" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 46097 } } peers { permanent_uuid: "c8ef706c3c2141aa895ff4a4b0b3a3da" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 44909 } } peers { permanent_uuid: "d17b19f91f0c44a48c760e9cbf12de0a" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 45077 } }
I20260929 03:18:06.571244 29213 raft_consensus.cc:374] T 46ed7d7f48644de581f100003efeed67 P c8ef706c3c2141aa895ff4a4b0b3a3da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:18:06.571266 29213 raft_consensus.cc:729] T 46ed7d7f48644de581f100003efeed67 P c8ef706c3c2141aa895ff4a4b0b3a3da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8ef706c3c2141aa895ff4a4b0b3a3da, State: Initialized, Role: FOLLOWER
I20260929 03:18:06.571313 29213 consensus_queue.cc:260] T 46ed7d7f48644de581f100003efeed67 P c8ef706c3c2141aa895ff4a4b0b3a3da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bb9788606a844aeac11ef74e3475047" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 46097 } } peers { permanent_uuid: "c8ef706c3c2141aa895ff4a4b0b3a3da" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 44909 } } peers { permanent_uuid: "d17b19f91f0c44a48c760e9cbf12de0a" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 45077 } }
I20260929 03:18:06.571403 29213 ts_tablet_manager.cc:1428] T 46ed7d7f48644de581f100003efeed67 P c8ef706c3c2141aa895ff4a4b0b3a3da: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260929 03:18:06.572006 29216 tablet_bootstrap.cc:492] T 46ed7d7f48644de581f100003efeed67 P 8bb9788606a844aeac11ef74e3475047: No bootstrap required, opened a new log
I20260929 03:18:06.572050 29216 ts_tablet_manager.cc:1397] T 46ed7d7f48644de581f100003efeed67 P 8bb9788606a844aeac11ef74e3475047: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260929 03:18:06.572218 29216 raft_consensus.cc:348] T 46ed7d7f48644de581f100003efeed67 P 8bb9788606a844aeac11ef74e3475047 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bb9788606a844aeac11ef74e3475047" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 46097 } } peers { permanent_uuid: "c8ef706c3c2141aa895ff4a4b0b3a3da" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 44909 } } peers { permanent_uuid: "d17b19f91f0c44a48c760e9cbf12de0a" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 45077 } }
I20260929 03:18:06.572273 29216 raft_consensus.cc:374] T 46ed7d7f48644de581f100003efeed67 P 8bb9788606a844aeac11ef74e3475047 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:18:06.572285 29216 raft_consensus.cc:729] T 46ed7d7f48644de581f100003efeed67 P 8bb9788606a844aeac11ef74e3475047 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8bb9788606a844aeac11ef74e3475047, State: Initialized, Role: FOLLOWER
I20260929 03:18:06.572340 29216 consensus_queue.cc:260] T 46ed7d7f48644de581f100003efeed67 P 8bb9788606a844aeac11ef74e3475047 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bb9788606a844aeac11ef74e3475047" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 46097 } } peers { permanent_uuid: "c8ef706c3c2141aa895ff4a4b0b3a3da" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 44909 } } peers { permanent_uuid: "d17b19f91f0c44a48c760e9cbf12de0a" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 45077 } }
I20260929 03:18:06.572448 29216 ts_tablet_manager.cc:1428] T 46ed7d7f48644de581f100003efeed67 P 8bb9788606a844aeac11ef74e3475047: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260929 03:18:06.620677 29220 raft_consensus.cc:482] T 46ed7d7f48644de581f100003efeed67 P 8bb9788606a844aeac11ef74e3475047 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:18:06.620775 29220 raft_consensus.cc:504] T 46ed7d7f48644de581f100003efeed67 P 8bb9788606a844aeac11ef74e3475047 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bb9788606a844aeac11ef74e3475047" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 46097 } } peers { permanent_uuid: "c8ef706c3c2141aa895ff4a4b0b3a3da" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 44909 } } peers { permanent_uuid: "d17b19f91f0c44a48c760e9cbf12de0a" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 45077 } }
I20260929 03:18:06.621111 29220 leader_election.cc:288] T 46ed7d7f48644de581f100003efeed67 P 8bb9788606a844aeac11ef74e3475047 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c8ef706c3c2141aa895ff4a4b0b3a3da (127.20.178.66:44909), d17b19f91f0c44a48c760e9cbf12de0a (127.20.178.67:45077)
I20260929 03:18:06.625082 29210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:06.621396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.65:37831 (local address 127.20.178.67:45077)
0929 03:18:06.621443 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0929 03:18:06.621446 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:18:06.621647 (+ 201us) server_negotiation.cc:407] Connection header received
0929 03:18:06.621758 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:18:06.621759 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:18:06.621768 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:18:06.621817 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:18:06.621987 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:06.622702 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:06.624242 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:06.624605 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:06.624745 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:06.624754 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0929 03:18:06.624960 (+ 206us) server_negotiation.cc:299] Negotiation successful
0929 03:18:06.625009 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260929 03:18:06.625097 29295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:06.621288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.67:45077 (local address 127.20.178.65:37831)
0929 03:18:06.621565 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:06.621576 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 03:18:06.621677 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:18:06.621823 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:18:06.621824 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:18:06.621827 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:18:06.621907 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:06.621909 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:06.622786 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:06.622787 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:06.624122 (+ 1335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:06.624125 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:06.624746 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:06.624747 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:06.624826 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:06.624830 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 03:18:06.624932 (+ 102us) client_negotiation.cc:240] Negotiation successful
0929 03:18:06.624980 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":204,"mutex_wait_us":2,"thread_start_us":101,"threads_started":1}
I20260929 03:18:06.625167 29258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:06.621486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.65:53455 (local address 127.20.178.66:44909)
0929 03:18:06.621537 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0929 03:18:06.621539 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:18:06.621591 (+ 52us) server_negotiation.cc:407] Connection header received
0929 03:18:06.621664 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:18:06.621665 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:18:06.621675 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:18:06.621758 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:18:06.621953 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:06.622432 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:06.624224 (+ 1792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:06.624817 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:06.624884 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:06.624892 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0929 03:18:06.625050 (+ 158us) server_negotiation.cc:299] Negotiation successful
0929 03:18:06.625096 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260929 03:18:06.625377 29294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:06.621288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.66:44909 (local address 127.20.178.65:53455)
0929 03:18:06.621507 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:06.621518 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 03:18:06.621608 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:18:06.621775 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:18:06.621776 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:18:06.621778 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:18:06.621865 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:06.621868 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:06.622535 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:06.622535 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:06.624133 (+ 1598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:06.624135 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:06.624893 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:06.624894 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:06.624964 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:06.624967 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 03:18:06.625263 (+ 296us) client_negotiation.cc:240] Negotiation successful
0929 03:18:06.625300 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201,"mutex_wait_us":1,"thread_start_us":97,"threads_started":1}
I20260929 03:18:06.625415 29136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46ed7d7f48644de581f100003efeed67" candidate_uuid: "8bb9788606a844aeac11ef74e3475047" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d17b19f91f0c44a48c760e9cbf12de0a" is_pre_election: true
I20260929 03:18:06.625486 29136 raft_consensus.cc:2454] T 46ed7d7f48644de581f100003efeed67 P d17b19f91f0c44a48c760e9cbf12de0a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8bb9788606a844aeac11ef74e3475047 in term 0.
I20260929 03:18:06.625610 29001 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46ed7d7f48644de581f100003efeed67" candidate_uuid: "8bb9788606a844aeac11ef74e3475047" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8ef706c3c2141aa895ff4a4b0b3a3da" is_pre_election: true
I20260929 03:18:06.625687 29001 raft_consensus.cc:2454] T 46ed7d7f48644de581f100003efeed67 P c8ef706c3c2141aa895ff4a4b0b3a3da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8bb9788606a844aeac11ef74e3475047 in term 0.
I20260929 03:18:06.625654 28792 leader_election.cc:302] T 46ed7d7f48644de581f100003efeed67 P 8bb9788606a844aeac11ef74e3475047 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8bb9788606a844aeac11ef74e3475047, d17b19f91f0c44a48c760e9cbf12de0a; no voters:
I20260929 03:18:06.625765 29220 raft_consensus.cc:2781] T 46ed7d7f48644de581f100003efeed67 P 8bb9788606a844aeac11ef74e3475047 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 03:18:06.625790 29220 raft_consensus.cc:482] T 46ed7d7f48644de581f100003efeed67 P 8bb9788606a844aeac11ef74e3475047 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 03:18:06.625811 29220 raft_consensus.cc:3037] T 46ed7d7f48644de581f100003efeed67 P 8bb9788606a844aeac11ef74e3475047 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:18:06.626641 29220 raft_consensus.cc:504] T 46ed7d7f48644de581f100003efeed67 P 8bb9788606a844aeac11ef74e3475047 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bb9788606a844aeac11ef74e3475047" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 46097 } } peers { permanent_uuid: "c8ef706c3c2141aa895ff4a4b0b3a3da" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 44909 } } peers { permanent_uuid: "d17b19f91f0c44a48c760e9cbf12de0a" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 45077 } }
I20260929 03:18:06.626778 29220 leader_election.cc:288] T 46ed7d7f48644de581f100003efeed67 P 8bb9788606a844aeac11ef74e3475047 [CANDIDATE]: Term 1 election: Requested vote from peers c8ef706c3c2141aa895ff4a4b0b3a3da (127.20.178.66:44909), d17b19f91f0c44a48c760e9cbf12de0a (127.20.178.67:45077)
I20260929 03:18:06.626935 29001 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46ed7d7f48644de581f100003efeed67" candidate_uuid: "8bb9788606a844aeac11ef74e3475047" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8ef706c3c2141aa895ff4a4b0b3a3da"
I20260929 03:18:06.627003 29001 raft_consensus.cc:3037] T 46ed7d7f48644de581f100003efeed67 P c8ef706c3c2141aa895ff4a4b0b3a3da [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:18:06.626981 29136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46ed7d7f48644de581f100003efeed67" candidate_uuid: "8bb9788606a844aeac11ef74e3475047" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d17b19f91f0c44a48c760e9cbf12de0a"
I20260929 03:18:06.627053 29136 raft_consensus.cc:3037] T 46ed7d7f48644de581f100003efeed67 P d17b19f91f0c44a48c760e9cbf12de0a [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:18:06.627631 29001 raft_consensus.cc:2454] T 46ed7d7f48644de581f100003efeed67 P c8ef706c3c2141aa895ff4a4b0b3a3da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8bb9788606a844aeac11ef74e3475047 in term 1.
I20260929 03:18:06.627763 29136 raft_consensus.cc:2454] T 46ed7d7f48644de581f100003efeed67 P d17b19f91f0c44a48c760e9cbf12de0a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8bb9788606a844aeac11ef74e3475047 in term 1.
I20260929 03:18:06.627821 28793 leader_election.cc:302] T 46ed7d7f48644de581f100003efeed67 P 8bb9788606a844aeac11ef74e3475047 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8bb9788606a844aeac11ef74e3475047, c8ef706c3c2141aa895ff4a4b0b3a3da; no voters:
I20260929 03:18:06.628017 29220 raft_consensus.cc:2781] T 46ed7d7f48644de581f100003efeed67 P 8bb9788606a844aeac11ef74e3475047 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:18:06.628177 29220 raft_consensus.cc:686] T 46ed7d7f48644de581f100003efeed67 P 8bb9788606a844aeac11ef74e3475047 [term 1 LEADER]: Becoming Leader. State: Replica: 8bb9788606a844aeac11ef74e3475047, State: Running, Role: LEADER
I20260929 03:18:06.628243 29220 consensus_queue.cc:237] T 46ed7d7f48644de581f100003efeed67 P 8bb9788606a844aeac11ef74e3475047 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bb9788606a844aeac11ef74e3475047" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 46097 } } peers { permanent_uuid: "c8ef706c3c2141aa895ff4a4b0b3a3da" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 44909 } } peers { permanent_uuid: "d17b19f91f0c44a48c760e9cbf12de0a" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 45077 } }
I20260929 03:18:06.628923 28703 catalog_manager.cc:5495] T 46ed7d7f48644de581f100003efeed67 P 8bb9788606a844aeac11ef74e3475047 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8bb9788606a844aeac11ef74e3475047 (127.20.178.65). New cstate: current_term: 1 leader_uuid: "8bb9788606a844aeac11ef74e3475047" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bb9788606a844aeac11ef74e3475047" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 46097 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8ef706c3c2141aa895ff4a4b0b3a3da" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 44909 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d17b19f91f0c44a48c760e9cbf12de0a" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 45077 } health_report { overall_health: UNKNOWN } } }
I20260929 03:18:06.647580 29211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:06.645219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47580 (local address 127.20.178.65:46097)
0929 03:18:06.645264 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0929 03:18:06.645266 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:18:06.645275 (+ 9us) server_negotiation.cc:407] Connection header received
0929 03:18:06.645352 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:18:06.645353 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:18:06.645382 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:18:06.645462 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:18:06.645577 (+ 115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:06.646070 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:06.646824 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:06.647005 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:06.647085 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:06.647257 (+ 172us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:18:06.647362 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:18:06.647442 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0929 03:18:06.647479 (+ 37us) server_negotiation.cc:299] Negotiation successful
0929 03:18:06.647499 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260929 03:18:06.648535 29001 raft_consensus.cc:1261] T 46ed7d7f48644de581f100003efeed67 P c8ef706c3c2141aa895ff4a4b0b3a3da [term 1 FOLLOWER]: Refusing update from remote peer 8bb9788606a844aeac11ef74e3475047: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:18:06.648551 29136 raft_consensus.cc:1261] T 46ed7d7f48644de581f100003efeed67 P d17b19f91f0c44a48c760e9cbf12de0a [term 1 FOLLOWER]: Refusing update from remote peer 8bb9788606a844aeac11ef74e3475047: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:18:06.648834 29296 consensus_queue.cc:1035] T 46ed7d7f48644de581f100003efeed67 P 8bb9788606a844aeac11ef74e3475047 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8ef706c3c2141aa895ff4a4b0b3a3da" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 44909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:18:06.648969 29220 consensus_queue.cc:1035] T 46ed7d7f48644de581f100003efeed67 P 8bb9788606a844aeac11ef74e3475047 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d17b19f91f0c44a48c760e9cbf12de0a" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 45077 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:18:06.652071 28703 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57134:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:18:06.652172 28703 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57134:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:18:06.652846 28703 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b8fefe3633e6491689c51b85ebdc7336: Sending DeleteTablet for 3 replicas of tablet 46ed7d7f48644de581f100003efeed67
I20260929 03:18:06.653118 29121 tablet_service.cc:1514] Processing DeleteTablet for tablet 46ed7d7f48644de581f100003efeed67 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:18:06 UTC) from {username='oryx', principal='oryx/127.20.178.126@KRBTEST.COM'} at 127.0.0.1:38304
I20260929 03:18:06.653158 28981 tablet_service.cc:1514] Processing DeleteTablet for tablet 46ed7d7f48644de581f100003efeed67 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:18:06 UTC) from {username='oryx', principal='oryx/127.20.178.126@KRBTEST.COM'} at 127.0.0.1:33508
I20260929 03:18:06.653210 29276 tablet_replica.cc:330] T 46ed7d7f48644de581f100003efeed67 P d17b19f91f0c44a48c760e9cbf12de0a: stopping tablet replica
I20260929 03:18:06.653261 29276 raft_consensus.cc:2229] T 46ed7d7f48644de581f100003efeed67 P d17b19f91f0c44a48c760e9cbf12de0a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:18:06.653244 29278 tablet_replica.cc:330] T 46ed7d7f48644de581f100003efeed67 P c8ef706c3c2141aa895ff4a4b0b3a3da: stopping tablet replica
I20260929 03:18:06.653306 29276 raft_consensus.cc:2258] T 46ed7d7f48644de581f100003efeed67 P d17b19f91f0c44a48c760e9cbf12de0a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:18:06.653314 29278 raft_consensus.cc:2229] T 46ed7d7f48644de581f100003efeed67 P c8ef706c3c2141aa895ff4a4b0b3a3da [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:18:06.653355 29278 raft_consensus.cc:2258] T 46ed7d7f48644de581f100003efeed67 P c8ef706c3c2141aa895ff4a4b0b3a3da [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:18:06.653673 29278 ts_tablet_manager.cc:1905] T 46ed7d7f48644de581f100003efeed67 P c8ef706c3c2141aa895ff4a4b0b3a3da: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:18:06.653690 28841 tablet_service.cc:1514] Processing DeleteTablet for tablet 46ed7d7f48644de581f100003efeed67 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:18:06 UTC) from {username='oryx', principal='oryx/127.20.178.126@KRBTEST.COM'} at 127.0.0.1:47574
I20260929 03:18:06.653759 29277 tablet_replica.cc:330] T 46ed7d7f48644de581f100003efeed67 P 8bb9788606a844aeac11ef74e3475047: stopping tablet replica
I20260929 03:18:06.653803 29277 raft_consensus.cc:2229] T 46ed7d7f48644de581f100003efeed67 P 8bb9788606a844aeac11ef74e3475047 [term 1 LEADER]: Raft consensus shutting down.
I20260929 03:18:06.653877 29277 raft_consensus.cc:2258] T 46ed7d7f48644de581f100003efeed67 P 8bb9788606a844aeac11ef74e3475047 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:18:06.654018 28703 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57134:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 03:18:06.654137 28703 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 03:18:06.655381 29278 ts_tablet_manager.cc:1918] T 46ed7d7f48644de581f100003efeed67 P c8ef706c3c2141aa895ff4a4b0b3a3da: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:18:06.655432 29278 log.cc:1196] T 46ed7d7f48644de581f100003efeed67 P c8ef706c3c2141aa895ff4a4b0b3a3da: Deleting WAL directory at /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651852476208-21193-0/minicluster-data/ts-1/wal/wals/46ed7d7f48644de581f100003efeed67
I20260929 03:18:06.655628 29278 ts_tablet_manager.cc:1939] T 46ed7d7f48644de581f100003efeed67 P c8ef706c3c2141aa895ff4a4b0b3a3da: Deleting consensus metadata
I20260929 03:18:06.656121 28689 catalog_manager.cc:4841] TS c8ef706c3c2141aa895ff4a4b0b3a3da (127.20.178.66:44909): tablet 46ed7d7f48644de581f100003efeed67 (table test-table [id=1f2cc505dfaf40f4b3ce463cf119a8d8]) successfully deleted
I20260929 03:18:06.656183 28981 tablet_service.cc:1467] Processing CreateTablet for tablet 3310e2e5b12b4743a7f4ff3e58469404 (DEFAULT_TABLE table=test-table [id=96a56484f33c4b6ebdf204a4ca2a870d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:18:06.656294 28981 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3310e2e5b12b4743a7f4ff3e58469404. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:18:06.657368 29213 tablet_bootstrap.cc:492] T 3310e2e5b12b4743a7f4ff3e58469404 P c8ef706c3c2141aa895ff4a4b0b3a3da: Bootstrap starting.
I20260929 03:18:06.657557 29121 tablet_service.cc:1467] Processing CreateTablet for tablet 3310e2e5b12b4743a7f4ff3e58469404 (DEFAULT_TABLE table=test-table [id=96a56484f33c4b6ebdf204a4ca2a870d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:18:06.657648 29121 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3310e2e5b12b4743a7f4ff3e58469404. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:18:06.657685 29213 tablet_bootstrap.cc:654] T 3310e2e5b12b4743a7f4ff3e58469404 P c8ef706c3c2141aa895ff4a4b0b3a3da: Neither blocks nor log segments found. Creating new log.
I20260929 03:18:06.658618 29212 tablet_bootstrap.cc:492] T 3310e2e5b12b4743a7f4ff3e58469404 P d17b19f91f0c44a48c760e9cbf12de0a: Bootstrap starting.
I20260929 03:18:06.658792 28841 tablet_service.cc:1467] Processing CreateTablet for tablet 3310e2e5b12b4743a7f4ff3e58469404 (DEFAULT_TABLE table=test-table [id=96a56484f33c4b6ebdf204a4ca2a870d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:18:06.658885 28841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3310e2e5b12b4743a7f4ff3e58469404. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:18:06.658941 29212 tablet_bootstrap.cc:654] T 3310e2e5b12b4743a7f4ff3e58469404 P d17b19f91f0c44a48c760e9cbf12de0a: Neither blocks nor log segments found. Creating new log.
I20260929 03:18:06.659852 29216 tablet_bootstrap.cc:492] T 3310e2e5b12b4743a7f4ff3e58469404 P 8bb9788606a844aeac11ef74e3475047: Bootstrap starting.
I20260929 03:18:06.660259 29216 tablet_bootstrap.cc:654] T 3310e2e5b12b4743a7f4ff3e58469404 P 8bb9788606a844aeac11ef74e3475047: Neither blocks nor log segments found. Creating new log.
I20260929 03:18:06.660490 29276 ts_tablet_manager.cc:1905] T 46ed7d7f48644de581f100003efeed67 P d17b19f91f0c44a48c760e9cbf12de0a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:18:06.661136 29216 tablet_bootstrap.cc:492] T 3310e2e5b12b4743a7f4ff3e58469404 P 8bb9788606a844aeac11ef74e3475047: No bootstrap required, opened a new log
I20260929 03:18:06.661181 29216 ts_tablet_manager.cc:1397] T 3310e2e5b12b4743a7f4ff3e58469404 P 8bb9788606a844aeac11ef74e3475047: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260929 03:18:06.661347 29216 raft_consensus.cc:348] T 3310e2e5b12b4743a7f4ff3e58469404 P 8bb9788606a844aeac11ef74e3475047 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bb9788606a844aeac11ef74e3475047" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 46097 } } peers { permanent_uuid: "c8ef706c3c2141aa895ff4a4b0b3a3da" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 44909 } } peers { permanent_uuid: "d17b19f91f0c44a48c760e9cbf12de0a" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 45077 } }
I20260929 03:18:06.661424 29216 raft_consensus.cc:374] T 3310e2e5b12b4743a7f4ff3e58469404 P 8bb9788606a844aeac11ef74e3475047 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:18:06.661448 29216 raft_consensus.cc:729] T 3310e2e5b12b4743a7f4ff3e58469404 P 8bb9788606a844aeac11ef74e3475047 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8bb9788606a844aeac11ef74e3475047, State: Initialized, Role: FOLLOWER
I20260929 03:18:06.661487 29216 consensus_queue.cc:260] T 3310e2e5b12b4743a7f4ff3e58469404 P 8bb9788606a844aeac11ef74e3475047 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bb9788606a844aeac11ef74e3475047" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 46097 } } peers { permanent_uuid: "c8ef706c3c2141aa895ff4a4b0b3a3da" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 44909 } } peers { permanent_uuid: "d17b19f91f0c44a48c760e9cbf12de0a" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 45077 } }
I20260929 03:18:06.661573 29216 ts_tablet_manager.cc:1428] T 3310e2e5b12b4743a7f4ff3e58469404 P 8bb9788606a844aeac11ef74e3475047: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260929 03:18:06.661985 29276 ts_tablet_manager.cc:1918] T 46ed7d7f48644de581f100003efeed67 P d17b19f91f0c44a48c760e9cbf12de0a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:18:06.662038 29276 log.cc:1196] T 46ed7d7f48644de581f100003efeed67 P d17b19f91f0c44a48c760e9cbf12de0a: Deleting WAL directory at /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651852476208-21193-0/minicluster-data/ts-2/wal/wals/46ed7d7f48644de581f100003efeed67
I20260929 03:18:06.662297 29276 ts_tablet_manager.cc:1939] T 46ed7d7f48644de581f100003efeed67 P d17b19f91f0c44a48c760e9cbf12de0a: Deleting consensus metadata
I20260929 03:18:06.662638 28688 catalog_manager.cc:4841] TS d17b19f91f0c44a48c760e9cbf12de0a (127.20.178.67:45077): tablet 46ed7d7f48644de581f100003efeed67 (table test-table [id=1f2cc505dfaf40f4b3ce463cf119a8d8]) successfully deleted
I20260929 03:18:06.665306 29213 tablet_bootstrap.cc:492] T 3310e2e5b12b4743a7f4ff3e58469404 P c8ef706c3c2141aa895ff4a4b0b3a3da: No bootstrap required, opened a new log
I20260929 03:18:06.665354 29213 ts_tablet_manager.cc:1397] T 3310e2e5b12b4743a7f4ff3e58469404 P c8ef706c3c2141aa895ff4a4b0b3a3da: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.000s
I20260929 03:18:06.665455 29213 raft_consensus.cc:348] T 3310e2e5b12b4743a7f4ff3e58469404 P c8ef706c3c2141aa895ff4a4b0b3a3da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bb9788606a844aeac11ef74e3475047" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 46097 } } peers { permanent_uuid: "c8ef706c3c2141aa895ff4a4b0b3a3da" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 44909 } } peers { permanent_uuid: "d17b19f91f0c44a48c760e9cbf12de0a" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 45077 } }
I20260929 03:18:06.665512 29213 raft_consensus.cc:374] T 3310e2e5b12b4743a7f4ff3e58469404 P c8ef706c3c2141aa895ff4a4b0b3a3da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:18:06.665529 29213 raft_consensus.cc:729] T 3310e2e5b12b4743a7f4ff3e58469404 P c8ef706c3c2141aa895ff4a4b0b3a3da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8ef706c3c2141aa895ff4a4b0b3a3da, State: Initialized, Role: FOLLOWER
I20260929 03:18:06.665594 29213 consensus_queue.cc:260] T 3310e2e5b12b4743a7f4ff3e58469404 P c8ef706c3c2141aa895ff4a4b0b3a3da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bb9788606a844aeac11ef74e3475047" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 46097 } } peers { permanent_uuid: "c8ef706c3c2141aa895ff4a4b0b3a3da" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 44909 } } peers { permanent_uuid: "d17b19f91f0c44a48c760e9cbf12de0a" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 45077 } }
I20260929 03:18:06.665688 29213 ts_tablet_manager.cc:1428] T 3310e2e5b12b4743a7f4ff3e58469404 P c8ef706c3c2141aa895ff4a4b0b3a3da: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260929 03:18:06.665769 29277 ts_tablet_manager.cc:1905] T 46ed7d7f48644de581f100003efeed67 P 8bb9788606a844aeac11ef74e3475047: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:18:06.666572 29212 tablet_bootstrap.cc:492] T 3310e2e5b12b4743a7f4ff3e58469404 P d17b19f91f0c44a48c760e9cbf12de0a: No bootstrap required, opened a new log
I20260929 03:18:06.666620 29212 ts_tablet_manager.cc:1397] T 3310e2e5b12b4743a7f4ff3e58469404 P d17b19f91f0c44a48c760e9cbf12de0a: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.001s
I20260929 03:18:06.666729 29212 raft_consensus.cc:348] T 3310e2e5b12b4743a7f4ff3e58469404 P d17b19f91f0c44a48c760e9cbf12de0a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bb9788606a844aeac11ef74e3475047" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 46097 } } peers { permanent_uuid: "c8ef706c3c2141aa895ff4a4b0b3a3da" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 44909 } } peers { permanent_uuid: "d17b19f91f0c44a48c760e9cbf12de0a" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 45077 } }
I20260929 03:18:06.666793 29212 raft_consensus.cc:374] T 3310e2e5b12b4743a7f4ff3e58469404 P d17b19f91f0c44a48c760e9cbf12de0a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:18:06.666812 29212 raft_consensus.cc:729] T 3310e2e5b12b4743a7f4ff3e58469404 P d17b19f91f0c44a48c760e9cbf12de0a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d17b19f91f0c44a48c760e9cbf12de0a, State: Initialized, Role: FOLLOWER
I20260929 03:18:06.666849 29212 consensus_queue.cc:260] T 3310e2e5b12b4743a7f4ff3e58469404 P d17b19f91f0c44a48c760e9cbf12de0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bb9788606a844aeac11ef74e3475047" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 46097 } } peers { permanent_uuid: "c8ef706c3c2141aa895ff4a4b0b3a3da" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 44909 } } peers { permanent_uuid: "d17b19f91f0c44a48c760e9cbf12de0a" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 45077 } }
I20260929 03:18:06.666923 29212 ts_tablet_manager.cc:1428] T 3310e2e5b12b4743a7f4ff3e58469404 P d17b19f91f0c44a48c760e9cbf12de0a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260929 03:18:06.667266 29277 ts_tablet_manager.cc:1918] T 46ed7d7f48644de581f100003efeed67 P 8bb9788606a844aeac11ef74e3475047: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:18:06.667338 29277 log.cc:1196] T 46ed7d7f48644de581f100003efeed67 P 8bb9788606a844aeac11ef74e3475047: Deleting WAL directory at /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651852476208-21193-0/minicluster-data/ts-0/wal/wals/46ed7d7f48644de581f100003efeed67
I20260929 03:18:06.667528 29277 ts_tablet_manager.cc:1939] T 46ed7d7f48644de581f100003efeed67 P 8bb9788606a844aeac11ef74e3475047: Deleting consensus metadata
I20260929 03:18:06.667961 28688 catalog_manager.cc:4841] TS 8bb9788606a844aeac11ef74e3475047 (127.20.178.65:46097): tablet 46ed7d7f48644de581f100003efeed67 (table test-table [id=1f2cc505dfaf40f4b3ce463cf119a8d8]) successfully deleted
I20260929 03:18:06.778512 29222 raft_consensus.cc:482] T 3310e2e5b12b4743a7f4ff3e58469404 P d17b19f91f0c44a48c760e9cbf12de0a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:18:06.778620 29222 raft_consensus.cc:504] T 3310e2e5b12b4743a7f4ff3e58469404 P d17b19f91f0c44a48c760e9cbf12de0a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bb9788606a844aeac11ef74e3475047" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 46097 } } peers { permanent_uuid: "c8ef706c3c2141aa895ff4a4b0b3a3da" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 44909 } } peers { permanent_uuid: "d17b19f91f0c44a48c760e9cbf12de0a" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 45077 } }
I20260929 03:18:06.778831 29222 leader_election.cc:288] T 3310e2e5b12b4743a7f4ff3e58469404 P d17b19f91f0c44a48c760e9cbf12de0a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8bb9788606a844aeac11ef74e3475047 (127.20.178.65:46097), c8ef706c3c2141aa895ff4a4b0b3a3da (127.20.178.66:44909)
I20260929 03:18:06.779024 28859 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3310e2e5b12b4743a7f4ff3e58469404" candidate_uuid: "d17b19f91f0c44a48c760e9cbf12de0a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8bb9788606a844aeac11ef74e3475047" is_pre_election: true
I20260929 03:18:06.779065 29001 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3310e2e5b12b4743a7f4ff3e58469404" candidate_uuid: "d17b19f91f0c44a48c760e9cbf12de0a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8ef706c3c2141aa895ff4a4b0b3a3da" is_pre_election: true
I20260929 03:18:06.779156 28859 raft_consensus.cc:2454] T 3310e2e5b12b4743a7f4ff3e58469404 P 8bb9788606a844aeac11ef74e3475047 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d17b19f91f0c44a48c760e9cbf12de0a in term 0.
I20260929 03:18:06.779170 29001 raft_consensus.cc:2454] T 3310e2e5b12b4743a7f4ff3e58469404 P c8ef706c3c2141aa895ff4a4b0b3a3da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d17b19f91f0c44a48c760e9cbf12de0a in term 0.
I20260929 03:18:06.779321 29073 leader_election.cc:302] T 3310e2e5b12b4743a7f4ff3e58469404 P d17b19f91f0c44a48c760e9cbf12de0a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c8ef706c3c2141aa895ff4a4b0b3a3da, d17b19f91f0c44a48c760e9cbf12de0a; no voters:
I20260929 03:18:06.779433 29222 raft_consensus.cc:2781] T 3310e2e5b12b4743a7f4ff3e58469404 P d17b19f91f0c44a48c760e9cbf12de0a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 03:18:06.779465 29222 raft_consensus.cc:482] T 3310e2e5b12b4743a7f4ff3e58469404 P d17b19f91f0c44a48c760e9cbf12de0a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 03:18:06.779484 29222 raft_consensus.cc:3037] T 3310e2e5b12b4743a7f4ff3e58469404 P d17b19f91f0c44a48c760e9cbf12de0a [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:18:06.780329 29222 raft_consensus.cc:504] T 3310e2e5b12b4743a7f4ff3e58469404 P d17b19f91f0c44a48c760e9cbf12de0a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bb9788606a844aeac11ef74e3475047" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 46097 } } peers { permanent_uuid: "c8ef706c3c2141aa895ff4a4b0b3a3da" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 44909 } } peers { permanent_uuid: "d17b19f91f0c44a48c760e9cbf12de0a" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 45077 } }
I20260929 03:18:06.780460 29222 leader_election.cc:288] T 3310e2e5b12b4743a7f4ff3e58469404 P d17b19f91f0c44a48c760e9cbf12de0a [CANDIDATE]: Term 1 election: Requested vote from peers 8bb9788606a844aeac11ef74e3475047 (127.20.178.65:46097), c8ef706c3c2141aa895ff4a4b0b3a3da (127.20.178.66:44909)
I20260929 03:18:06.780660 29001 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3310e2e5b12b4743a7f4ff3e58469404" candidate_uuid: "d17b19f91f0c44a48c760e9cbf12de0a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8ef706c3c2141aa895ff4a4b0b3a3da"
I20260929 03:18:06.780695 28859 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3310e2e5b12b4743a7f4ff3e58469404" candidate_uuid: "d17b19f91f0c44a48c760e9cbf12de0a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8bb9788606a844aeac11ef74e3475047"
I20260929 03:18:06.780761 28859 raft_consensus.cc:3037] T 3310e2e5b12b4743a7f4ff3e58469404 P 8bb9788606a844aeac11ef74e3475047 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:18:06.780763 29001 raft_consensus.cc:3037] T 3310e2e5b12b4743a7f4ff3e58469404 P c8ef706c3c2141aa895ff4a4b0b3a3da [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:18:06.781386 29001 raft_consensus.cc:2454] T 3310e2e5b12b4743a7f4ff3e58469404 P c8ef706c3c2141aa895ff4a4b0b3a3da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d17b19f91f0c44a48c760e9cbf12de0a in term 1.
I20260929 03:18:06.781418 28859 raft_consensus.cc:2454] T 3310e2e5b12b4743a7f4ff3e58469404 P 8bb9788606a844aeac11ef74e3475047 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d17b19f91f0c44a48c760e9cbf12de0a in term 1.
I20260929 03:18:06.781611 29072 leader_election.cc:302] T 3310e2e5b12b4743a7f4ff3e58469404 P d17b19f91f0c44a48c760e9cbf12de0a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 8bb9788606a844aeac11ef74e3475047, c8ef706c3c2141aa895ff4a4b0b3a3da, d17b19f91f0c44a48c760e9cbf12de0a; no voters:
I20260929 03:18:06.781713 29222 raft_consensus.cc:2781] T 3310e2e5b12b4743a7f4ff3e58469404 P d17b19f91f0c44a48c760e9cbf12de0a [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:18:06.781817 29222 raft_consensus.cc:686] T 3310e2e5b12b4743a7f4ff3e58469404 P d17b19f91f0c44a48c760e9cbf12de0a [term 1 LEADER]: Becoming Leader. State: Replica: d17b19f91f0c44a48c760e9cbf12de0a, State: Running, Role: LEADER
I20260929 03:18:06.781863 29222 consensus_queue.cc:237] T 3310e2e5b12b4743a7f4ff3e58469404 P d17b19f91f0c44a48c760e9cbf12de0a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bb9788606a844aeac11ef74e3475047" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 46097 } } peers { permanent_uuid: "c8ef706c3c2141aa895ff4a4b0b3a3da" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 44909 } } peers { permanent_uuid: "d17b19f91f0c44a48c760e9cbf12de0a" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 45077 } }
I20260929 03:18:06.782375 28703 catalog_manager.cc:5495] T 3310e2e5b12b4743a7f4ff3e58469404 P d17b19f91f0c44a48c760e9cbf12de0a reported cstate change: term changed from 0 to 1, leader changed from <none> to d17b19f91f0c44a48c760e9cbf12de0a (127.20.178.67). New cstate: current_term: 1 leader_uuid: "d17b19f91f0c44a48c760e9cbf12de0a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bb9788606a844aeac11ef74e3475047" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 46097 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8ef706c3c2141aa895ff4a4b0b3a3da" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 44909 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d17b19f91f0c44a48c760e9cbf12de0a" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 45077 } health_report { overall_health: HEALTHY } } }
I20260929 03:18:06.816560 29210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:06.814370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38312 (local address 127.20.178.67:45077)
0929 03:18:06.814417 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0929 03:18:06.814419 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:18:06.814430 (+ 11us) server_negotiation.cc:407] Connection header received
0929 03:18:06.814505 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:18:06.814506 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:18:06.814516 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:18:06.814560 (+ 44us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:18:06.814682 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:06.815163 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:06.815885 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:06.816100 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:06.816176 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:06.816319 (+ 143us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:18:06.816352 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:18:06.816400 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0929 03:18:06.816450 (+ 50us) server_negotiation.cc:299] Negotiation successful
0929 03:18:06.816470 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260929 03:18:06.820986 29258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:06.818780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.67:39275 (local address 127.20.178.66:44909)
0929 03:18:06.818819 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0929 03:18:06.818820 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 03:18:06.818929 (+ 109us) server_negotiation.cc:407] Connection header received
0929 03:18:06.818979 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:18:06.818979 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:18:06.818989 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:18:06.819035 (+ 46us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:18:06.819146 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:06.819598 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:06.820312 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:06.820480 (+ 168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:06.820528 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:06.820739 (+ 211us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:18:06.820774 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:18:06.820817 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0929 03:18:06.820914 (+ 97us) server_negotiation.cc:299] Negotiation successful
0929 03:18:06.820931 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260929 03:18:06.821020 29310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:06.818688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.66:44909 (local address 127.20.178.67:39275)
0929 03:18:06.818869 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:06.818891 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 03:18:06.818941 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:18:06.819044 (+ 103us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:18:06.819044 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:18:06.819045 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 03:18:06.819109 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:06.819110 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:06.819660 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:06.819661 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:06.820240 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:06.820242 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:06.820541 (+ 299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:06.820541 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:06.820592 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:06.820605 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 03:18:06.820856 (+ 251us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 03:18:06.820859 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 03:18:06.820912 (+ 53us) client_negotiation.cc:240] Negotiation successful
0929 03:18:06.820954 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":87,"threads_started":1}
I20260929 03:18:06.822927 28859 raft_consensus.cc:1261] T 3310e2e5b12b4743a7f4ff3e58469404 P 8bb9788606a844aeac11ef74e3475047 [term 1 FOLLOWER]: Refusing update from remote peer d17b19f91f0c44a48c760e9cbf12de0a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:18:06.822926 29001 raft_consensus.cc:1261] T 3310e2e5b12b4743a7f4ff3e58469404 P c8ef706c3c2141aa895ff4a4b0b3a3da [term 1 FOLLOWER]: Refusing update from remote peer d17b19f91f0c44a48c760e9cbf12de0a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:18:06.823128 29306 consensus_queue.cc:1035] T 3310e2e5b12b4743a7f4ff3e58469404 P d17b19f91f0c44a48c760e9cbf12de0a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8bb9788606a844aeac11ef74e3475047" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 46097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:18:06.823251 29222 consensus_queue.cc:1035] T 3310e2e5b12b4743a7f4ff3e58469404 P d17b19f91f0c44a48c760e9cbf12de0a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8ef706c3c2141aa895ff4a4b0b3a3da" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 44909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:18:06.829783 29271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:06.827334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.67:45077 (local address 127.20.178.66:47517)
0929 03:18:06.827396 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:06.827407 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 03:18:06.827448 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:18:06.827670 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:18:06.827671 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:18:06.827672 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 03:18:06.827717 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:06.827719 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:06.828395 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:06.828396 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:06.829016 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:06.829018 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:06.829414 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:06.829415 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:06.829468 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:06.829475 (+ 7us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 03:18:06.829632 (+ 157us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 03:18:06.829634 (+ 2us) client_negotiation.cc:769] Sending connection context
0929 03:18:06.829672 (+ 38us) client_negotiation.cc:240] Negotiation successful
0929 03:18:06.829717 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260929 03:18:06.829782 29210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:06.827519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.66:47517 (local address 127.20.178.67:45077)
0929 03:18:06.827560 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0929 03:18:06.827562 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:18:06.827570 (+ 8us) server_negotiation.cc:407] Connection header received
0929 03:18:06.827590 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:18:06.827591 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:18:06.827601 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:18:06.827658 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:18:06.827812 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:06.828316 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:06.829114 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:06.829347 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:06.829405 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:06.829547 (+ 142us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:18:06.829577 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:18:06.829630 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0929 03:18:06.829685 (+ 55us) server_negotiation.cc:299] Negotiation successful
0929 03:18:06.829710 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260929 03:18:06.835704 28704 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57134:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:18:06.835821 28704 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57134:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:18:06.836465 28704 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b8fefe3633e6491689c51b85ebdc7336: Sending DeleteTablet for 3 replicas of tablet 3310e2e5b12b4743a7f4ff3e58469404
I20260929 03:18:06.836742 28841 tablet_service.cc:1514] Processing DeleteTablet for tablet 3310e2e5b12b4743a7f4ff3e58469404 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:18:06 UTC) from {username='oryx', principal='oryx/127.20.178.126@KRBTEST.COM'} at 127.0.0.1:47574
I20260929 03:18:06.836779 28981 tablet_service.cc:1514] Processing DeleteTablet for tablet 3310e2e5b12b4743a7f4ff3e58469404 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:18:06 UTC) from {username='oryx', principal='oryx/127.20.178.126@KRBTEST.COM'} at 127.0.0.1:33508
I20260929 03:18:06.836843 29121 tablet_service.cc:1514] Processing DeleteTablet for tablet 3310e2e5b12b4743a7f4ff3e58469404 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:18:06 UTC) from {username='oryx', principal='oryx/127.20.178.126@KRBTEST.COM'} at 127.0.0.1:38304
I20260929 03:18:06.836867 29278 tablet_replica.cc:330] T 3310e2e5b12b4743a7f4ff3e58469404 P c8ef706c3c2141aa895ff4a4b0b3a3da: stopping tablet replica
I20260929 03:18:06.836928 29278 raft_consensus.cc:2229] T 3310e2e5b12b4743a7f4ff3e58469404 P c8ef706c3c2141aa895ff4a4b0b3a3da [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:18:06.836951 29277 tablet_replica.cc:330] T 3310e2e5b12b4743a7f4ff3e58469404 P 8bb9788606a844aeac11ef74e3475047: stopping tablet replica
I20260929 03:18:06.836969 29278 raft_consensus.cc:2258] T 3310e2e5b12b4743a7f4ff3e58469404 P c8ef706c3c2141aa895ff4a4b0b3a3da [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:18:06.837013 29277 raft_consensus.cc:2229] T 3310e2e5b12b4743a7f4ff3e58469404 P 8bb9788606a844aeac11ef74e3475047 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:18:06.837054 29277 raft_consensus.cc:2258] T 3310e2e5b12b4743a7f4ff3e58469404 P 8bb9788606a844aeac11ef74e3475047 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:18:06.837133 29276 tablet_replica.cc:330] T 3310e2e5b12b4743a7f4ff3e58469404 P d17b19f91f0c44a48c760e9cbf12de0a: stopping tablet replica
I20260929 03:18:06.837193 29276 raft_consensus.cc:2229] T 3310e2e5b12b4743a7f4ff3e58469404 P d17b19f91f0c44a48c760e9cbf12de0a [term 1 LEADER]: Raft consensus shutting down.
I20260929 03:18:06.837199 21193 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuoaHbX/build/release/bin/kudu with pid 28776
I20260929 03:18:06.837251 29276 raft_consensus.cc:2258] T 3310e2e5b12b4743a7f4ff3e58469404 P d17b19f91f0c44a48c760e9cbf12de0a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:18:06.837361 29278 ts_tablet_manager.cc:1905] T 3310e2e5b12b4743a7f4ff3e58469404 P c8ef706c3c2141aa895ff4a4b0b3a3da: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:18:06.837909 29276 ts_tablet_manager.cc:1905] T 3310e2e5b12b4743a7f4ff3e58469404 P d17b19f91f0c44a48c760e9cbf12de0a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:18:06.839298 29278 ts_tablet_manager.cc:1918] T 3310e2e5b12b4743a7f4ff3e58469404 P c8ef706c3c2141aa895ff4a4b0b3a3da: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 03:18:06.839353 29278 log.cc:1196] T 3310e2e5b12b4743a7f4ff3e58469404 P c8ef706c3c2141aa895ff4a4b0b3a3da: Deleting WAL directory at /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651852476208-21193-0/minicluster-data/ts-1/wal/wals/3310e2e5b12b4743a7f4ff3e58469404
I20260929 03:18:06.839568 29278 ts_tablet_manager.cc:1939] T 3310e2e5b12b4743a7f4ff3e58469404 P c8ef706c3c2141aa895ff4a4b0b3a3da: Deleting consensus metadata
I20260929 03:18:06.839726 29276 ts_tablet_manager.cc:1918] T 3310e2e5b12b4743a7f4ff3e58469404 P d17b19f91f0c44a48c760e9cbf12de0a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 03:18:06.839778 29276 log.cc:1196] T 3310e2e5b12b4743a7f4ff3e58469404 P d17b19f91f0c44a48c760e9cbf12de0a: Deleting WAL directory at /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651852476208-21193-0/minicluster-data/ts-2/wal/wals/3310e2e5b12b4743a7f4ff3e58469404
I20260929 03:18:06.839980 29276 ts_tablet_manager.cc:1939] T 3310e2e5b12b4743a7f4ff3e58469404 P d17b19f91f0c44a48c760e9cbf12de0a: Deleting consensus metadata
I20260929 03:18:06.839973 28689 catalog_manager.cc:4841] TS c8ef706c3c2141aa895ff4a4b0b3a3da (127.20.178.66:44909): tablet 3310e2e5b12b4743a7f4ff3e58469404 (table test-table [id=96a56484f33c4b6ebdf204a4ca2a870d]) successfully deleted
I20260929 03:18:06.840485 28688 catalog_manager.cc:4841] TS d17b19f91f0c44a48c760e9cbf12de0a (127.20.178.67:45077): tablet 3310e2e5b12b4743a7f4ff3e58469404 (table test-table [id=96a56484f33c4b6ebdf204a4ca2a870d]) successfully deleted
W20260929 03:18:06.844553 28688 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.20.178.65:46097 (error 108)
I20260929 03:18:06.844764 21193 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuoaHbX/build/release/bin/kudu with pid 28916
I20260929 03:18:06.849434 29315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:06.849210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.65:46097 (local address 127.0.0.1:47586)
0929 03:18:06.849347 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:06.849399 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.178.65:46097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":88,"threads_started":1}
W20260929 03:18:06.849593 28688 catalog_manager.cc:4584] TS 8bb9788606a844aeac11ef74e3475047 (127.20.178.65:46097): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3310e2e5b12b4743a7f4ff3e58469404: Network error: Client connection negotiation failed: client connection to 127.20.178.65:46097: connect: Connection refused (error 111)
I20260929 03:18:06.850706 21193 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuoaHbX/build/release/bin/kudu with pid 29056
I20260929 03:18:06.856220 21193 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuoaHbX/build/release/bin/kudu with pid 28672
2026-09-29T03:18:06Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (2992 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790651852476208-21193-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:18:06 dist-test-slave-c9qh krb5kdc[29318](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:18:06 dist-test-slave-c9qh krb5kdc[29318](info): set up 2 sockets
Sep 29 03:18:06 dist-test-slave-c9qh krb5kdc[29318](info): commencing operation
krb5kdc: starting...
W20260929 03:18:07.886113 21193 mini_kdc.cc:121] Time spent starting KDC: real 1.006s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790651852476208-21193-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790651852476208-21193-0/krb5kdc/test-user.keytab.
Sep 29 03:18:07 dist-test-slave-c9qh krb5kdc[29318](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651887, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:18:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:18:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.178.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.178.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.178.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.126.keytab.
Entry for principal kudu/127.20.178.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.126.keytab.
I20260929 03:18:08.005478 21193 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuoaHbX/build/release/bin/kudu
/tmp/dist-test-taskuoaHbX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790651852476208-21193-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790651852476208-21193-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790651852476208-21193-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790651852476208-21193-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.178.126:38117
--webserver_interface=127.20.178.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.126.keytab
--principal=kudu/127.20.178.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.178.84:43865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.178.126:38117
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790651852476208-21193-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790651852476208-21193-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790651852476208-21193-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:18:08.085484 29331 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:18:08.085690 29331 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:18:08.085712 29331 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:18:08.086840 29331 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:18:08.086882 29331 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:18:08.086895 29331 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:18:08.086908 29331 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:18:08.086961 29331 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:18:08.088398 29331 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.178.84:43865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790651852476208-21193-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790651852476208-21193-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.178.126:38117
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.126.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.178.126:38117
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790651852476208-21193-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.178.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29331
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790651852476208-21193-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:18:08.088678 29331 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:18:08.088905 29331 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:18:08.093596 29336 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:18:08.093586 29339 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:18:08.093585 29337 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:18:08.093905 29331 server_base.cc:1034] running on GCE node
I20260929 03:18:08.094090 29331 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:18:08.094318 29331 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:18:08.095482 29331 hybrid_clock.cc:648] HybridClock initialized: now 1790651888095469 us; error 26 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-29T03:18:08Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1243 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651852476208-21193-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:18:08 dist-test-slave-c9qh krb5kdc[29344](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:18:08 dist-test-slave-c9qh krb5kdc[29344](info): set up 2 sockets
Sep 29 03:18:08 dist-test-slave-c9qh krb5kdc[29344](info): commencing operation
krb5kdc: starting...
W20260929 03:18:09.133258 21193 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651852476208-21193-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651852476208-21193-0/krb5kdc/test-user.keytab.
Sep 29 03:18:09 dist-test-slave-c9qh krb5kdc[29344](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651889, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:18:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:18:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.178.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.178.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.178.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.126.keytab.
Entry for principal kudu/127.20.178.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.126.keytab.
I20260929 03:18:09.247983 21193 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuoaHbX/build/release/bin/kudu
/tmp/dist-test-taskuoaHbX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651852476208-21193-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651852476208-21193-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651852476208-21193-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651852476208-21193-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.178.126:44981
--webserver_interface=127.20.178.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.126.keytab
--principal=kudu/127.20.178.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.178.84:44235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.178.126:44981
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651852476208-21193-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651852476208-21193-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651852476208-21193-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:18:09.330015 29357 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:18:09.330209 29357 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:18:09.330231 29357 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:18:09.331699 29357 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:18:09.331749 29357 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:18:09.331764 29357 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:18:09.331777 29357 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:18:09.331789 29357 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:18:09.333551 29357 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.178.84:44235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651852476208-21193-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651852476208-21193-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.178.126:44981
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.126.keytab
--principal=kudu/127.20.178.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.178.126:44981
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651852476208-21193-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.178.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29357
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651852476208-21193-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:18:09.333804 29357 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:18:09.334049 29357 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:18:09.338702 29362 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:18:09.338766 29363 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:18:09.338701 29365 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:18:09.339008 29357 server_base.cc:1034] running on GCE node
I20260929 03:18:09.339214 29357 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:18:09.339437 29357 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:18:09.340621 29357 hybrid_clock.cc:648] HybridClock initialized: now 1790651889340572 us; error 28 us; skew 500 ppm
Sep 29 03:18:09 dist-test-slave-c9qh krb5kdc[29344](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651889, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:18:09.348863 29357 init.cc:377] Logged in from keytab as kudu/127.20.178.126@KRBTEST.COM (short username kudu)
I20260929 03:18:09.349357 29357 webserver.cc:460] Webserver started at http://127.20.178.126:36469/ using document root <none> and password file <none>
I20260929 03:18:09.349566 29357 fs_manager.cc:359] Metadata directory not provided
I20260929 03:18:09.349608 29357 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:18:09.349709 29357 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:18:09.350713 29357 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651852476208-21193-0/minicluster-data/master-0/data/instance:
uuid: "0b487997ff314e23b66a0523d8957d22"
format_stamp: "Formatted at 2026-09-29 03:18:09 on dist-test-slave-c9qh"
server_key: "b280b2c5cc797652c9d4fc884b8ea223"
server_key_iv: "8aa26393441e0df740891a0864d890f9"
server_key_version: "encryptionkey@0"
I20260929 03:18:09.351006 29357 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651852476208-21193-0/minicluster-data/master-0/wal/instance:
uuid: "0b487997ff314e23b66a0523d8957d22"
format_stamp: "Formatted at 2026-09-29 03:18:09 on dist-test-slave-c9qh"
server_key: "b280b2c5cc797652c9d4fc884b8ea223"
server_key_iv: "8aa26393441e0df740891a0864d890f9"
server_key_version: "encryptionkey@0"
I20260929 03:18:09.352276 29357 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 03:18:09.353183 29372 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:18:09.353361 29357 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260929 03:18:09.353451 29357 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651852476208-21193-0/minicluster-data/master-0/data,/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651852476208-21193-0/minicluster-data/master-0/wal
uuid: "0b487997ff314e23b66a0523d8957d22"
format_stamp: "Formatted at 2026-09-29 03:18:09 on dist-test-slave-c9qh"
server_key: "b280b2c5cc797652c9d4fc884b8ea223"
server_key_iv: "8aa26393441e0df740891a0864d890f9"
server_key_version: "encryptionkey@0"
I20260929 03:18:09.353514 29357 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651852476208-21193-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651852476208-21193-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651852476208-21193-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:18:09.367280 29357 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:18:09.369077 29357 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:18:09.369185 29357 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:18:09.372926 29357 rpc_server.cc:307] RPC server started. Bound to: 127.20.178.126:44981
I20260929 03:18:09.372954 29424 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.178.126:44981 every 8 connection(s)
I20260929 03:18:09.373301 29357 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651852476208-21193-0/minicluster-data/master-0/data/info.pb
I20260929 03:18:09.374169 29425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:18:09.376243 29425 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b487997ff314e23b66a0523d8957d22: Bootstrap starting.
I20260929 03:18:09.376770 29425 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0b487997ff314e23b66a0523d8957d22: Neither blocks nor log segments found. Creating new log.
I20260929 03:18:09.376997 29425 log.cc:824] T 00000000000000000000000000000000 P 0b487997ff314e23b66a0523d8957d22: Log is configured to *not* fsync() on all Append() calls
I20260929 03:18:09.377610 29425 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b487997ff314e23b66a0523d8957d22: No bootstrap required, opened a new log
I20260929 03:18:09.379086 29425 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0b487997ff314e23b66a0523d8957d22 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b487997ff314e23b66a0523d8957d22" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 44981 } }
I20260929 03:18:09.379213 29425 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0b487997ff314e23b66a0523d8957d22 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:18:09.379235 29425 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0b487997ff314e23b66a0523d8957d22 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b487997ff314e23b66a0523d8957d22, State: Initialized, Role: FOLLOWER
I20260929 03:18:09.379365 29425 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0b487997ff314e23b66a0523d8957d22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b487997ff314e23b66a0523d8957d22" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 44981 } }
I20260929 03:18:09.379447 29425 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0b487997ff314e23b66a0523d8957d22 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:18:09.379472 29425 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0b487997ff314e23b66a0523d8957d22 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:18:09.379498 29425 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0b487997ff314e23b66a0523d8957d22 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:18:09.379963 29425 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0b487997ff314e23b66a0523d8957d22 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b487997ff314e23b66a0523d8957d22" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 44981 } }
I20260929 03:18:09.380041 29425 leader_election.cc:302] T 00000000000000000000000000000000 P 0b487997ff314e23b66a0523d8957d22 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0b487997ff314e23b66a0523d8957d22; no voters:
I20260929 03:18:09.380190 29425 leader_election.cc:288] T 00000000000000000000000000000000 P 0b487997ff314e23b66a0523d8957d22 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:18:09.380231 29428 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0b487997ff314e23b66a0523d8957d22 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:18:09.380350 29428 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0b487997ff314e23b66a0523d8957d22 [term 1 LEADER]: Becoming Leader. State: Replica: 0b487997ff314e23b66a0523d8957d22, State: Running, Role: LEADER
I20260929 03:18:09.380447 29425 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0b487997ff314e23b66a0523d8957d22 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:18:09.380426 29428 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0b487997ff314e23b66a0523d8957d22 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b487997ff314e23b66a0523d8957d22" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 44981 } }
I20260929 03:18:09.380884 29429 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b487997ff314e23b66a0523d8957d22 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0b487997ff314e23b66a0523d8957d22. Latest consensus state: current_term: 1 leader_uuid: "0b487997ff314e23b66a0523d8957d22" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b487997ff314e23b66a0523d8957d22" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 44981 } } }
I20260929 03:18:09.380990 29429 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b487997ff314e23b66a0523d8957d22 [sys.catalog]: This master's current role is: LEADER
I20260929 03:18:09.381024 29430 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b487997ff314e23b66a0523d8957d22 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0b487997ff314e23b66a0523d8957d22" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b487997ff314e23b66a0523d8957d22" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 44981 } } }
I20260929 03:18:09.381150 29430 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b487997ff314e23b66a0523d8957d22 [sys.catalog]: This master's current role is: LEADER
I20260929 03:18:09.381253 29436 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:18:09.381871 29436 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:18:09.383111 21193 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuoaHbX/build/release/bin/kudu as pid 29357
I20260929 03:18:09.383220 21193 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651852476208-21193-0/minicluster-data/master-0/wal/instance
I20260929 03:18:09.383358 29436 catalog_manager.cc:1347] Generated new cluster ID: 3ad5349dae9749b9ba958f67db50cfe4
I20260929 03:18:09.383407 29436 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:18:09.383410 21193 external_mini_cluster.cc:1428] Setting key 98aa98efe6535c78e3fed6a261a48809
Sep 29 03:18:09 dist-test-slave-c9qh krb5kdc[29344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651889, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.178.126@KRBTEST.COM
I20260929 03:18:09.391984 29436 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:18:09.392531 29436 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:18:09.399042 29436 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0b487997ff314e23b66a0523d8957d22: Generated new TSK 0
I20260929 03:18:09.399217 29436 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 03:18:09.399436 29448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:09.383841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53100 (local address 127.20.178.126:44981)
0929 03:18:09.383989 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0929 03:18:09.383998 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 03:18:09.384060 (+ 62us) server_negotiation.cc:407] Connection header received
0929 03:18:09.384207 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:18:09.384213 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:18:09.384269 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:18:09.384398 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:18:09.385178 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:09.386531 (+ 1353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:09.387605 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:09.388186 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:09.388279 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:09.391051 (+ 2772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:18:09.391055 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:18:09.391055 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:18:09.391067 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:18:09.397562 (+ 6495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:18:09.397963 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:18:09.397966 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:18:09.397966 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:18:09.398067 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:18:09.398278 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:18:09.398278 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:18:09.398279 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:18:09.398542 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:18:09.398698 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0929 03:18:09.399034 (+ 336us) server_negotiation.cc:299] Negotiation successful
0929 03:18:09.399218 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102,"thread_start_us":79,"threads_started":1}
WARNING: no policy specified for kudu/127.20.178.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.178.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.178.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.65.keytab.
Entry for principal kudu/127.20.178.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.65.keytab.
I20260929 03:18:09.425073 21193 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuoaHbX/build/release/bin/kudu
/tmp/dist-test-taskuoaHbX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651852476208-21193-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651852476208-21193-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651852476208-21193-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651852476208-21193-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.178.65:0
--local_ip_for_outbound_sockets=127.20.178.65
--webserver_interface=127.20.178.65
--webserver_port=0
--tserver_master_addrs=127.20.178.126:44981
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.65.keytab
--principal=kudu/127.20.178.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.178.84:44235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651852476208-21193-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651852476208-21193-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651852476208-21193-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:18:09.506207 29451 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:18:09.506399 29451 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:18:09.506422 29451 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:18:09.507579 29451 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:18:09.507628 29451 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:18:09.507654 29451 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.178.65
I20260929 03:18:09.509233 29451 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.178.84:44235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651852476208-21193-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651852476208-21193-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.65.keytab
--principal=kudu/127.20.178.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.178.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651852476208-21193-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.178.65
--webserver_port=0
--tserver_master_addrs=127.20.178.126:44981
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29451
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.178.65
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651852476208-21193-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:18:09.509470 29451 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:18:09.509712 29451 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 03:18:09.514695 29451 server_base.cc:1034] running on GCE node
W20260929 03:18:09.514695 29457 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:18:09.514755 29456 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:18:09.514708 29459 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:18:09.515106 29451 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:18:09.515311 29451 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:18:09.516472 29451 hybrid_clock.cc:648] HybridClock initialized: now 1790651889516462 us; error 28 us; skew 500 ppm
Sep 29 03:18:09 dist-test-slave-c9qh krb5kdc[29344](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651889, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:18:09.518989 29451 init.cc:377] Logged in from keytab as kudu/127.20.178.65@KRBTEST.COM (short username kudu)
I20260929 03:18:09.519462 29451 webserver.cc:460] Webserver started at http://127.20.178.65:35625/ using document root <none> and password file <none>
I20260929 03:18:09.519713 29451 fs_manager.cc:359] Metadata directory not provided
I20260929 03:18:09.519784 29451 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:18:09.519941 29451 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:18:09.521055 29451 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651852476208-21193-0/minicluster-data/ts-0/data/instance:
uuid: "4099362e38dd4571be411d4f137a9bf5"
format_stamp: "Formatted at 2026-09-29 03:18:09 on dist-test-slave-c9qh"
server_key: "a9d387e8566210a2fbb6bee6d0929ff1"
server_key_iv: "1d55d4c49aa8496fa33056f20d7ce20a"
server_key_version: "encryptionkey@0"
I20260929 03:18:09.521456 29451 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651852476208-21193-0/minicluster-data/ts-0/wal/instance:
uuid: "4099362e38dd4571be411d4f137a9bf5"
format_stamp: "Formatted at 2026-09-29 03:18:09 on dist-test-slave-c9qh"
server_key: "a9d387e8566210a2fbb6bee6d0929ff1"
server_key_iv: "1d55d4c49aa8496fa33056f20d7ce20a"
server_key_version: "encryptionkey@0"
I20260929 03:18:09.522866 29451 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260929 03:18:09.523839 29466 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:18:09.524061 29451 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 03:18:09.524163 29451 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651852476208-21193-0/minicluster-data/ts-0/data,/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651852476208-21193-0/minicluster-data/ts-0/wal
uuid: "4099362e38dd4571be411d4f137a9bf5"
format_stamp: "Formatted at 2026-09-29 03:18:09 on dist-test-slave-c9qh"
server_key: "a9d387e8566210a2fbb6bee6d0929ff1"
server_key_iv: "1d55d4c49aa8496fa33056f20d7ce20a"
server_key_version: "encryptionkey@0"
I20260929 03:18:09.524246 29451 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651852476208-21193-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651852476208-21193-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651852476208-21193-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:18:09.554448 29451 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:18:09.556757 29451 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:18:09.556931 29451 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:18:09.557219 29451 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:18:09.557547 29451 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:18:09.557581 29451 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:18:09.557643 29451 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:18:09.557670 29451 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:18:09.563360 29451 rpc_server.cc:307] RPC server started. Bound to: 127.20.178.65:46815
I20260929 03:18:09.563402 29579 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.178.65:46815 every 8 connection(s)
I20260929 03:18:09.563740 29451 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651852476208-21193-0/minicluster-data/ts-0/data/info.pb
Sep 29 03:18:09 dist-test-slave-c9qh krb5kdc[29344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651889, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.65@KRBTEST.COM for kudu/127.20.178.126@KRBTEST.COM
I20260929 03:18:09.569811 21193 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuoaHbX/build/release/bin/kudu as pid 29451
I20260929 03:18:09.569900 21193 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651852476208-21193-0/minicluster-data/ts-0/wal/instance
I20260929 03:18:09.570118 21193 external_mini_cluster.cc:1428] Setting key 83f9adc27c483a88d19c94ccfab8b5db
I20260929 03:18:09.573110 29448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:09.564487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.65:43125 (local address 127.20.178.126:44981)
0929 03:18:09.564545 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0929 03:18:09.564547 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:18:09.564775 (+ 228us) server_negotiation.cc:407] Connection header received
0929 03:18:09.565252 (+ 477us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:18:09.565253 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:18:09.565262 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:18:09.565316 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:18:09.566210 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:09.566698 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:09.567654 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:09.567860 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:09.567912 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:09.570245 (+ 2333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:18:09.570247 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:18:09.570247 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:18:09.570253 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:18:09.571666 (+ 1413us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:18:09.572087 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:18:09.572088 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:18:09.572090 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:18:09.572148 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:18:09.572420 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:18:09.572420 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:18:09.572421 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:18:09.572532 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:18:09.572624 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0929 03:18:09.572931 (+ 307us) server_negotiation.cc:299] Negotiation successful
0929 03:18:09.573020 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260929 03:18:09.573297 29582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:09.564472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:44981 (local address 127.20.178.65:43125)
0929 03:18:09.564623 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:09.564691 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0929 03:18:09.565173 (+ 482us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:18:09.565371 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:18:09.565374 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:18:09.565934 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:18:09.566130 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:09.566135 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:09.566785 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:09.566786 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:09.567566 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:09.567568 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:09.567936 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:09.567937 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:09.568038 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:09.568573 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:18:09.568581 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:18:09.570107 (+ 1526us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:18:09.571764 (+ 1657us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:18:09.571771 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:18:09.571773 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:18:09.571999 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:18:09.572230 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:18:09.572231 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:18:09.572231 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:18:09.572345 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:18:09.572611 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:18:09.572618 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:18:09.572796 (+ 178us) client_negotiation.cc:769] Sending connection context
0929 03:18:09.572908 (+ 112us) client_negotiation.cc:240] Negotiation successful
0929 03:18:09.573060 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":73,"threads_started":1}
I20260929 03:18:09.573771 29580 heartbeater.cc:347] Connected to a master server at 127.20.178.126:44981
I20260929 03:18:09.573872 29580 heartbeater.cc:464] Registering TS with master...
I20260929 03:18:09.574075 29580 heartbeater.cc:511] Master 127.20.178.126:44981 requested a full tablet report, sending...
I20260929 03:18:09.574663 29389 ts_manager.cc:194] Registered new tserver with Master: 4099362e38dd4571be411d4f137a9bf5 (127.20.178.65:46815)
I20260929 03:18:09.575645 29389 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.178.65@KRBTEST.COM'} at 127.20.178.65:43125
WARNING: no policy specified for kudu/127.20.178.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.178.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.178.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.66.keytab.
Entry for principal kudu/127.20.178.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.66.keytab.
I20260929 03:18:09.595536 21193 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuoaHbX/build/release/bin/kudu
/tmp/dist-test-taskuoaHbX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651852476208-21193-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651852476208-21193-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651852476208-21193-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651852476208-21193-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.178.66:0
--local_ip_for_outbound_sockets=127.20.178.66
--webserver_interface=127.20.178.66
--webserver_port=0
--tserver_master_addrs=127.20.178.126:44981
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.66.keytab
--principal=kudu/127.20.178.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.178.84:44235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651852476208-21193-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651852476208-21193-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651852476208-21193-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:18:09.673910 29585 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:18:09.674098 29585 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:18:09.674119 29585 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:18:09.675264 29585 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:18:09.675304 29585 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:18:09.675330 29585 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.178.66
I20260929 03:18:09.676741 29585 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.178.84:44235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651852476208-21193-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651852476208-21193-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.66.keytab
--principal=kudu/127.20.178.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.178.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651852476208-21193-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.178.66
--webserver_port=0
--tserver_master_addrs=127.20.178.126:44981
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29585
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.178.66
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651852476208-21193-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:18:09.676978 29585 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:18:09.677194 29585 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:18:09.682221 29590 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:18:09.682216 29593 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:18:09.682219 29591 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:18:09.682420 29585 server_base.cc:1034] running on GCE node
I20260929 03:18:09.682610 29585 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:18:09.682857 29585 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:18:09.684015 29585 hybrid_clock.cc:648] HybridClock initialized: now 1790651889684003 us; error 25 us; skew 500 ppm
Sep 29 03:18:09 dist-test-slave-c9qh krb5kdc[29344](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651889, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:18:09.686365 29585 init.cc:377] Logged in from keytab as kudu/127.20.178.66@KRBTEST.COM (short username kudu)
I20260929 03:18:09.686805 29585 webserver.cc:460] Webserver started at http://127.20.178.66:40973/ using document root <none> and password file <none>
I20260929 03:18:09.687038 29585 fs_manager.cc:359] Metadata directory not provided
I20260929 03:18:09.687104 29585 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:18:09.687255 29585 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:18:09.688257 29585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651852476208-21193-0/minicluster-data/ts-1/data/instance:
uuid: "4cd1656091f14a86919ae2ed3063dc00"
format_stamp: "Formatted at 2026-09-29 03:18:09 on dist-test-slave-c9qh"
server_key: "259a53b0c8a46ef3e3aa98c09997401a"
server_key_iv: "f5f09ff3f118aabc0b12d7160f84617f"
server_key_version: "encryptionkey@0"
I20260929 03:18:09.688604 29585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651852476208-21193-0/minicluster-data/ts-1/wal/instance:
uuid: "4cd1656091f14a86919ae2ed3063dc00"
format_stamp: "Formatted at 2026-09-29 03:18:09 on dist-test-slave-c9qh"
server_key: "259a53b0c8a46ef3e3aa98c09997401a"
server_key_iv: "f5f09ff3f118aabc0b12d7160f84617f"
server_key_version: "encryptionkey@0"
I20260929 03:18:09.689908 29585 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 03:18:09.690589 29600 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:18:09.690755 29585 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 03:18:09.690847 29585 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651852476208-21193-0/minicluster-data/ts-1/data,/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651852476208-21193-0/minicluster-data/ts-1/wal
uuid: "4cd1656091f14a86919ae2ed3063dc00"
format_stamp: "Formatted at 2026-09-29 03:18:09 on dist-test-slave-c9qh"
server_key: "259a53b0c8a46ef3e3aa98c09997401a"
server_key_iv: "f5f09ff3f118aabc0b12d7160f84617f"
server_key_version: "encryptionkey@0"
I20260929 03:18:09.690927 29585 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651852476208-21193-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651852476208-21193-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651852476208-21193-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:18:09.701261 29585 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:18:09.703081 29585 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:18:09.703238 29585 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:18:09.703446 29585 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:18:09.703743 29585 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:18:09.703795 29585 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:18:09.703835 29585 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:18:09.703864 29585 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:18:09.709615 29585 rpc_server.cc:307] RPC server started. Bound to: 127.20.178.66:33319
I20260929 03:18:09.709671 29713 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.178.66:33319 every 8 connection(s)
I20260929 03:18:09.710000 29585 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651852476208-21193-0/minicluster-data/ts-1/data/info.pb
Sep 29 03:18:09 dist-test-slave-c9qh krb5kdc[29344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651889, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.66@KRBTEST.COM for kudu/127.20.178.126@KRBTEST.COM
I20260929 03:18:09.719698 29448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:09.710763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.66:51637 (local address 127.20.178.126:44981)
0929 03:18:09.710831 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0929 03:18:09.710834 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:18:09.711012 (+ 178us) server_negotiation.cc:407] Connection header received
0929 03:18:09.711566 (+ 554us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:18:09.711568 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:18:09.711581 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:18:09.711640 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:18:09.712628 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:09.713346 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:09.714607 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:09.714828 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:09.714917 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:09.716981 (+ 2064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:18:09.716982 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:18:09.716983 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:18:09.716991 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:18:09.718338 (+ 1347us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:18:09.718797 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:18:09.718798 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:18:09.718798 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:18:09.718856 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:18:09.719160 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:18:09.719161 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:18:09.719161 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:18:09.719276 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:18:09.719328 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0929 03:18:09.719546 (+ 218us) server_negotiation.cc:299] Negotiation successful
0929 03:18:09.719604 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 03:18:09.719854 29716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:09.710724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:44981 (local address 127.20.178.66:51637)
0929 03:18:09.710890 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:09.710932 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 03:18:09.711443 (+ 511us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:18:09.711727 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:18:09.711730 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:18:09.712309 (+ 579us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:18:09.712497 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:09.712502 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:09.713450 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:09.713451 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:09.714389 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:09.714391 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:09.714927 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:09.714929 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:09.715051 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:09.715545 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:18:09.715553 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:18:09.716889 (+ 1336us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:18:09.718428 (+ 1539us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:18:09.718434 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:18:09.718450 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:18:09.718728 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:18:09.718945 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:18:09.718946 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:18:09.718947 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:18:09.719087 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:18:09.719335 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:18:09.719339 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:18:09.719444 (+ 105us) client_negotiation.cc:769] Sending connection context
0929 03:18:09.719525 (+ 81us) client_negotiation.cc:240] Negotiation successful
0929 03:18:09.719638 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":91,"threads_started":1}
I20260929 03:18:09.720052 21193 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuoaHbX/build/release/bin/kudu as pid 29585
I20260929 03:18:09.720177 21193 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651852476208-21193-0/minicluster-data/ts-1/wal/instance
I20260929 03:18:09.720378 21193 external_mini_cluster.cc:1428] Setting key 0fb0799ae28e44d9c980b2eab3bd6a30
I20260929 03:18:09.720381 29714 heartbeater.cc:347] Connected to a master server at 127.20.178.126:44981
I20260929 03:18:09.720476 29714 heartbeater.cc:464] Registering TS with master...
I20260929 03:18:09.720687 29714 heartbeater.cc:511] Master 127.20.178.126:44981 requested a full tablet report, sending...
I20260929 03:18:09.721040 29389 ts_manager.cc:194] Registered new tserver with Master: 4cd1656091f14a86919ae2ed3063dc00 (127.20.178.66:33319)
I20260929 03:18:09.721535 29389 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.178.66@KRBTEST.COM'} at 127.20.178.66:51637
WARNING: no policy specified for kudu/127.20.178.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.178.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.178.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.67.keytab.
Entry for principal kudu/127.20.178.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.67.keytab.
I20260929 03:18:09.743688 21193 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuoaHbX/build/release/bin/kudu
/tmp/dist-test-taskuoaHbX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651852476208-21193-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651852476208-21193-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651852476208-21193-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651852476208-21193-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.178.67:0
--local_ip_for_outbound_sockets=127.20.178.67
--webserver_interface=127.20.178.67
--webserver_port=0
--tserver_master_addrs=127.20.178.126:44981
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.67.keytab
--principal=kudu/127.20.178.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.178.84:44235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651852476208-21193-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651852476208-21193-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651852476208-21193-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:18:09.824401 29719 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:18:09.824601 29719 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:18:09.824625 29719 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:18:09.825733 29719 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:18:09.825768 29719 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:18:09.825793 29719 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.178.67
I20260929 03:18:09.827229 29719 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.178.84:44235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651852476208-21193-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651852476208-21193-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.67.keytab
--principal=kudu/127.20.178.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.178.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651852476208-21193-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.178.67
--webserver_port=0
--tserver_master_addrs=127.20.178.126:44981
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29719
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.178.67
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651852476208-21193-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:18:09.827414 29719 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:18:09.827632 29719 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:18:09.832427 29724 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:18:09.832458 29719 server_base.cc:1034] running on GCE node
W20260929 03:18:09.832594 29727 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:18:09.832468 29725 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:18:09.832890 29719 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:18:09.833072 29719 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:18:09.834223 29719 hybrid_clock.cc:648] HybridClock initialized: now 1790651889834210 us; error 27 us; skew 500 ppm
Sep 29 03:18:09 dist-test-slave-c9qh krb5kdc[29344](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651889, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:18:09.836534 29719 init.cc:377] Logged in from keytab as kudu/127.20.178.67@KRBTEST.COM (short username kudu)
I20260929 03:18:09.837035 29719 webserver.cc:460] Webserver started at http://127.20.178.67:39549/ using document root <none> and password file <none>
I20260929 03:18:09.837260 29719 fs_manager.cc:359] Metadata directory not provided
I20260929 03:18:09.837327 29719 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:18:09.837473 29719 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:18:09.838505 29719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651852476208-21193-0/minicluster-data/ts-2/data/instance:
uuid: "864babb0800d4871991b59bc820f2a7b"
format_stamp: "Formatted at 2026-09-29 03:18:09 on dist-test-slave-c9qh"
server_key: "9a8f92c62f2cf9b3024789457085de17"
server_key_iv: "0ec2eb0c16514ec191d88996c22f5bd9"
server_key_version: "encryptionkey@0"
I20260929 03:18:09.838871 29719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651852476208-21193-0/minicluster-data/ts-2/wal/instance:
uuid: "864babb0800d4871991b59bc820f2a7b"
format_stamp: "Formatted at 2026-09-29 03:18:09 on dist-test-slave-c9qh"
server_key: "9a8f92c62f2cf9b3024789457085de17"
server_key_iv: "0ec2eb0c16514ec191d88996c22f5bd9"
server_key_version: "encryptionkey@0"
I20260929 03:18:09.840134 29719 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 03:18:09.840845 29734 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:18:09.841009 29719 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260929 03:18:09.841125 29719 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651852476208-21193-0/minicluster-data/ts-2/data,/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651852476208-21193-0/minicluster-data/ts-2/wal
uuid: "864babb0800d4871991b59bc820f2a7b"
format_stamp: "Formatted at 2026-09-29 03:18:09 on dist-test-slave-c9qh"
server_key: "9a8f92c62f2cf9b3024789457085de17"
server_key_iv: "0ec2eb0c16514ec191d88996c22f5bd9"
server_key_version: "encryptionkey@0"
I20260929 03:18:09.841198 29719 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651852476208-21193-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651852476208-21193-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651852476208-21193-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:18:09.851073 29719 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:18:09.852948 29719 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:18:09.853120 29719 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:18:09.853336 29719 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:18:09.853644 29719 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:18:09.853696 29719 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:18:09.853734 29719 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:18:09.853765 29719 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:18:09.859850 29719 rpc_server.cc:307] RPC server started. Bound to: 127.20.178.67:44599
I20260929 03:18:09.859898 29847 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.178.67:44599 every 8 connection(s)
I20260929 03:18:09.860188 29719 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651852476208-21193-0/minicluster-data/ts-2/data/info.pb
Sep 29 03:18:09 dist-test-slave-c9qh krb5kdc[29344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651889, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.67@KRBTEST.COM for kudu/127.20.178.126@KRBTEST.COM
I20260929 03:18:09.868283 21193 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuoaHbX/build/release/bin/kudu as pid 29719
I20260929 03:18:09.868407 21193 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651852476208-21193-0/minicluster-data/ts-2/wal/instance
I20260929 03:18:09.868605 21193 external_mini_cluster.cc:1428] Setting key b0a5b8ec0506d399286da36f5aaff43d
I20260929 03:18:09.869345 29448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:09.860849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.67:56983 (local address 127.20.178.126:44981)
0929 03:18:09.860937 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0929 03:18:09.860939 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:18:09.861116 (+ 177us) server_negotiation.cc:407] Connection header received
0929 03:18:09.861629 (+ 513us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:18:09.861630 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:18:09.861643 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:18:09.861695 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:18:09.862579 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:09.863080 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:09.864150 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:09.864361 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:09.864431 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:09.866458 (+ 2027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:18:09.866459 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:18:09.866459 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:18:09.866466 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:18:09.867835 (+ 1369us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:18:09.868203 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:18:09.868205 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:18:09.868205 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:18:09.868267 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:18:09.868574 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:18:09.868575 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:18:09.868575 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:18:09.868773 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:18:09.868852 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0929 03:18:09.869179 (+ 327us) server_negotiation.cc:299] Negotiation successful
0929 03:18:09.869255 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 03:18:09.869551 29850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:09.860819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:44981 (local address 127.20.178.67:56983)
0929 03:18:09.860988 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:09.861032 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 03:18:09.861506 (+ 474us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:18:09.861759 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:18:09.861763 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:18:09.862300 (+ 537us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:18:09.862474 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:09.862478 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:09.863180 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:09.863181 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:09.864042 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:09.864044 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:09.864449 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:09.864449 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:09.864607 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:09.865056 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:18:09.865064 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:18:09.866373 (+ 1309us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:18:09.867906 (+ 1533us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:18:09.867911 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:18:09.867913 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:18:09.868117 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:18:09.868336 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:18:09.868337 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:18:09.868337 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:18:09.868508 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:18:09.868843 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:18:09.868849 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:18:09.869012 (+ 163us) client_negotiation.cc:769] Sending connection context
0929 03:18:09.869157 (+ 145us) client_negotiation.cc:240] Negotiation successful
0929 03:18:09.869308 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":81,"threads_started":1}
I20260929 03:18:09.869994 29848 heartbeater.cc:347] Connected to a master server at 127.20.178.126:44981
I20260929 03:18:09.870092 29848 heartbeater.cc:464] Registering TS with master...
I20260929 03:18:09.870258 29848 heartbeater.cc:511] Master 127.20.178.126:44981 requested a full tablet report, sending...
I20260929 03:18:09.870585 29389 ts_manager.cc:194] Registered new tserver with Master: 864babb0800d4871991b59bc820f2a7b (127.20.178.67:44599)
I20260929 03:18:09.870966 29389 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.178.67@KRBTEST.COM'} at 127.20.178.67:56983
I20260929 03:18:09.880352 21193 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.20.178.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.20.178.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.20.178.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651852476208-21193-0/krb5kdc/oryx_127.20.178.65.keytab.
Entry for principal oryx/127.20.178.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651852476208-21193-0/krb5kdc/oryx_127.20.178.65.keytab.
WARNING: no policy specified for oryx/127.20.178.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.20.178.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.20.178.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651852476208-21193-0/krb5kdc/oryx_127.20.178.66.keytab.
Entry for principal oryx/127.20.178.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651852476208-21193-0/krb5kdc/oryx_127.20.178.66.keytab.
WARNING: no policy specified for oryx/127.20.178.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.20.178.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.20.178.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651852476208-21193-0/krb5kdc/oryx_127.20.178.67.keytab.
Entry for principal oryx/127.20.178.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651852476208-21193-0/krb5kdc/oryx_127.20.178.67.keytab.
I20260929 03:18:09.947790 21193 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuoaHbX/build/release/bin/kudu with pid 29451
I20260929 03:18:09.952318 21193 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuoaHbX/build/release/bin/kudu with pid 29585
I20260929 03:18:09.956426 21193 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuoaHbX/build/release/bin/kudu with pid 29719
I20260929 03:18:09.960471 21193 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuoaHbX/build/release/bin/kudu with pid 29357
I20260929 03:18:09.965297 21193 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuoaHbX/build/release/bin/kudu
/tmp/dist-test-taskuoaHbX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651852476208-21193-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651852476208-21193-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651852476208-21193-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651852476208-21193-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.178.126:44981
--webserver_interface=127.20.178.126
--webserver_port=36469
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.126.keytab
--principal=kudu/127.20.178.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.178.84:44235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.178.126:44981
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651852476208-21193-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651852476208-21193-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651852476208-21193-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:18:10.045018 29857 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:18:10.045251 29857 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:18:10.045287 29857 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:18:10.046415 29857 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:18:10.046489 29857 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:18:10.046512 29857 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:18:10.046535 29857 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:18:10.046567 29857 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:18:10.048036 29857 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.178.84:44235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651852476208-21193-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651852476208-21193-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.178.126:44981
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.126.keytab
--principal=kudu/127.20.178.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.178.126:44981
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651852476208-21193-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.178.126
--webserver_port=36469
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29857
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651852476208-21193-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:18:10.048327 29857 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:18:10.048552 29857 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:18:10.053349 29863 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:18:10.053413 29865 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:18:10.053354 29862 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:18:10.053639 29857 server_base.cc:1034] running on GCE node
I20260929 03:18:10.053884 29857 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:18:10.054184 29857 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:18:10.055364 29857 hybrid_clock.cc:648] HybridClock initialized: now 1790651890055360 us; error 37 us; skew 500 ppm
Sep 29 03:18:10 dist-test-slave-c9qh krb5kdc[29344](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651890, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:18:10.057921 29857 init.cc:377] Logged in from keytab as kudu/127.20.178.126@KRBTEST.COM (short username kudu)
I20260929 03:18:10.058343 29857 webserver.cc:460] Webserver started at http://127.20.178.126:36469/ using document root <none> and password file <none>
I20260929 03:18:10.058594 29857 fs_manager.cc:359] Metadata directory not provided
I20260929 03:18:10.058686 29857 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:18:10.060073 29857 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 03:18:10.060796 29872 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:18:10.060968 29857 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260929 03:18:10.061076 29857 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651852476208-21193-0/minicluster-data/master-0/data,/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651852476208-21193-0/minicluster-data/master-0/wal
uuid: "0b487997ff314e23b66a0523d8957d22"
format_stamp: "Formatted at 2026-09-29 03:18:09 on dist-test-slave-c9qh"
server_key: "b280b2c5cc797652c9d4fc884b8ea223"
server_key_iv: "8aa26393441e0df740891a0864d890f9"
server_key_version: "encryptionkey@0"
I20260929 03:18:10.061430 29857 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651852476208-21193-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651852476208-21193-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651852476208-21193-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:18:10.068529 29857 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:18:10.070485 29857 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:18:10.070665 29857 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:18:10.074790 29857 rpc_server.cc:307] RPC server started. Bound to: 127.20.178.126:44981
I20260929 03:18:10.074818 29924 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.178.126:44981 every 8 connection(s)
I20260929 03:18:10.075207 29857 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651852476208-21193-0/minicluster-data/master-0/data/info.pb
I20260929 03:18:10.076576 29925 sys_catalog.cc:263] Verifying existing consensus state
I20260929 03:18:10.077224 29925 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b487997ff314e23b66a0523d8957d22: Bootstrap starting.
I20260929 03:18:10.079653 21193 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuoaHbX/build/release/bin/kudu as pid 29857
I20260929 03:18:10.079974 21193 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuoaHbX/build/release/bin/kudu
/tmp/dist-test-taskuoaHbX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651852476208-21193-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651852476208-21193-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651852476208-21193-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651852476208-21193-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.178.65:46815
--local_ip_for_outbound_sockets=127.20.178.65
--tserver_master_addrs=127.20.178.126:44981
--webserver_port=35625
--webserver_interface=127.20.178.65
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.65.keytab
--principal=kudu/127.20.178.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.178.84:44235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.20.178.65
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651852476208-21193-0/krb5kdc/oryx_127.20.178.65.keytab with env {KRB5CCNAME=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651852476208-21193-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651852476208-21193-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651852476208-21193-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 03:18:10.085716 29925 log.cc:824] T 00000000000000000000000000000000 P 0b487997ff314e23b66a0523d8957d22: Log is configured to *not* fsync() on all Append() calls
I20260929 03:18:10.087282 29925 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b487997ff314e23b66a0523d8957d22: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 03:18:10.087568 29925 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b487997ff314e23b66a0523d8957d22: Bootstrap complete.
I20260929 03:18:10.089843 29925 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0b487997ff314e23b66a0523d8957d22 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b487997ff314e23b66a0523d8957d22" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 44981 } }
I20260929 03:18:10.090076 29925 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0b487997ff314e23b66a0523d8957d22 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b487997ff314e23b66a0523d8957d22, State: Initialized, Role: FOLLOWER
I20260929 03:18:10.090252 29925 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0b487997ff314e23b66a0523d8957d22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b487997ff314e23b66a0523d8957d22" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 44981 } }
I20260929 03:18:10.090328 29925 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0b487997ff314e23b66a0523d8957d22 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:18:10.090379 29925 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0b487997ff314e23b66a0523d8957d22 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:18:10.090428 29925 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0b487997ff314e23b66a0523d8957d22 [term 1 FOLLOWER]: Advancing to term 2
I20260929 03:18:10.091355 29925 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0b487997ff314e23b66a0523d8957d22 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b487997ff314e23b66a0523d8957d22" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 44981 } }
I20260929 03:18:10.091471 29925 leader_election.cc:302] T 00000000000000000000000000000000 P 0b487997ff314e23b66a0523d8957d22 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0b487997ff314e23b66a0523d8957d22; no voters:
I20260929 03:18:10.091667 29925 leader_election.cc:288] T 00000000000000000000000000000000 P 0b487997ff314e23b66a0523d8957d22 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 03:18:10.091699 29929 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0b487997ff314e23b66a0523d8957d22 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 03:18:10.091813 29929 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0b487997ff314e23b66a0523d8957d22 [term 2 LEADER]: Becoming Leader. State: Replica: 0b487997ff314e23b66a0523d8957d22, State: Running, Role: LEADER
I20260929 03:18:10.091898 29929 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0b487997ff314e23b66a0523d8957d22 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b487997ff314e23b66a0523d8957d22" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 44981 } }
I20260929 03:18:10.091925 29925 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0b487997ff314e23b66a0523d8957d22 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:18:10.092159 29931 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b487997ff314e23b66a0523d8957d22 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "0b487997ff314e23b66a0523d8957d22" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b487997ff314e23b66a0523d8957d22" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 44981 } } }
I20260929 03:18:10.092245 29931 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b487997ff314e23b66a0523d8957d22 [sys.catalog]: This master's current role is: LEADER
I20260929 03:18:10.092480 29930 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b487997ff314e23b66a0523d8957d22 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0b487997ff314e23b66a0523d8957d22. Latest consensus state: current_term: 2 leader_uuid: "0b487997ff314e23b66a0523d8957d22" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b487997ff314e23b66a0523d8957d22" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 44981 } } }
I20260929 03:18:10.092617 29930 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b487997ff314e23b66a0523d8957d22 [sys.catalog]: This master's current role is: LEADER
I20260929 03:18:10.092792 29933 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:18:10.093326 29933 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:18:10.093559 29933 catalog_manager.cc:1259] Loaded cluster ID: 3ad5349dae9749b9ba958f67db50cfe4
I20260929 03:18:10.093812 29933 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:18:10.094601 29933 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:18:10.094789 29933 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 0b487997ff314e23b66a0523d8957d22: Loaded TSK: 0
I20260929 03:18:10.095000 29933 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 03:18:10.182314 29927 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:18:10.182512 29927 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:18:10.182535 29927 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:18:10.183686 29927 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:18:10.183730 29927 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:18:10.183756 29927 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.178.65
I20260929 03:18:10.185242 29927 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.178.84:44235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651852476208-21193-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651852476208-21193-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651852476208-21193-0/krb5kdc/oryx_127.20.178.65.keytab
--principal=oryx/127.20.178.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.178.65:46815
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651852476208-21193-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.178.65
--webserver_port=35625
--tserver_master_addrs=127.20.178.126:44981
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29927
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.178.65
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651852476208-21193-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:18:10.185467 29927 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:18:10.185695 29927 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:18:10.190478 29954 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:18:10.190483 29952 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:18:10.190474 29951 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:18:10.190713 29927 server_base.cc:1034] running on GCE node
I20260929 03:18:10.190954 29927 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:18:10.191184 29927 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:18:10.192349 29927 hybrid_clock.cc:648] HybridClock initialized: now 1790651890192336 us; error 31 us; skew 500 ppm
Sep 29 03:18:10 dist-test-slave-c9qh krb5kdc[29344](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651890, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.178.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:18:10.194998 29927 init.cc:377] Logged in from keytab as oryx/127.20.178.65@KRBTEST.COM (short username oryx)
I20260929 03:18:10.195456 29927 webserver.cc:460] Webserver started at http://127.20.178.65:35625/ using document root <none> and password file <none>
I20260929 03:18:10.195679 29927 fs_manager.cc:359] Metadata directory not provided
I20260929 03:18:10.195760 29927 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:18:10.197185 29927 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260929 03:18:10.197840 29961 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:18:10.198014 29927 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260929 03:18:10.198112 29927 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651852476208-21193-0/minicluster-data/ts-0/data,/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651852476208-21193-0/minicluster-data/ts-0/wal
uuid: "4099362e38dd4571be411d4f137a9bf5"
format_stamp: "Formatted at 2026-09-29 03:18:09 on dist-test-slave-c9qh"
server_key: "a9d387e8566210a2fbb6bee6d0929ff1"
server_key_iv: "1d55d4c49aa8496fa33056f20d7ce20a"
server_key_version: "encryptionkey@0"
I20260929 03:18:10.198446 29927 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651852476208-21193-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651852476208-21193-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651852476208-21193-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:18:10.211510 29927 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:18:10.213865 29927 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:18:10.214075 29927 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:18:10.214310 29927 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:18:10.214617 29927 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:18:10.214668 29927 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:18:10.214700 29927 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:18:10.214731 29927 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:18:10.220906 29927 rpc_server.cc:307] RPC server started. Bound to: 127.20.178.65:46815
I20260929 03:18:10.220952 30074 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.178.65:46815 every 8 connection(s)
I20260929 03:18:10.221278 29927 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651852476208-21193-0/minicluster-data/ts-0/data/info.pb
I20260929 03:18:10.225435 21193 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuoaHbX/build/release/bin/kudu as pid 29927
I20260929 03:18:10.225751 21193 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuoaHbX/build/release/bin/kudu
/tmp/dist-test-taskuoaHbX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651852476208-21193-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651852476208-21193-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651852476208-21193-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651852476208-21193-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.178.66:33319
--local_ip_for_outbound_sockets=127.20.178.66
--tserver_master_addrs=127.20.178.126:44981
--webserver_port=40973
--webserver_interface=127.20.178.66
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.66.keytab
--principal=kudu/127.20.178.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.178.84:44235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.20.178.66
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651852476208-21193-0/krb5kdc/oryx_127.20.178.66.keytab with env {KRB5CCNAME=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651852476208-21193-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651852476208-21193-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651852476208-21193-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 03:18:10 dist-test-slave-c9qh krb5kdc[29344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.178.65@KRBTEST.COM for oryx/127.20.178.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:18:10 dist-test-slave-c9qh krb5kdc[29344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.178.65@KRBTEST.COM for oryx/127.20.178.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:18:10.233492 30077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:10.221937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:44981 (local address 127.20.178.65:36971)
0929 03:18:10.222147 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:10.222194 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 03:18:10.222828 (+ 634us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:18:10.225232 (+ 2404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:18:10.225235 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:18:10.225908 (+ 673us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:18:10.226135 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:10.226140 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:10.227456 (+ 1316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:10.227458 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:10.228734 (+ 1276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:10.228737 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:10.229571 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:10.229572 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:10.229733 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:10.230351 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:18:10.230361 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:18:10.233257 (+ 2896us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.178.126:44981: Server oryx/127.20.178.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":115,"threads_started":1}
W20260929 03:18:10.233664 30078 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:18:10.222065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.65:36971 (local address 127.20.178.126:44981)
0929 03:18:10.224798 (+ 2733us) server_negotiation.cc:206] Beginning negotiation
0929 03:18:10.224807 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 03:18:10.224843 (+ 36us) server_negotiation.cc:407] Connection header received
0929 03:18:10.224973 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:18:10.224979 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:18:10.225032 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:18:10.225171 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:18:10.226254 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:10.227342 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:10.228823 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:10.229466 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:10.229572 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:10.233396 (+ 3824us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.178.65:36971: BlockingRecv error: recv got EOF from 127.20.178.65:36971 (error 108)
Metrics: {"server-negotiator.queue_time_us":2681,"thread_start_us":68,"threads_started":1}
W20260929 03:18:10.233743 30075 heartbeater.cc:644] Failed to heartbeat to 127.20.178.126:44981 (0 consecutive failures): Not authorized: Failed to ping master at 127.20.178.126:44981: Client connection negotiation failed: client connection to 127.20.178.126:44981: Server oryx/127.20.178.126@KRBTEST.COM not found in Kerberos database
Sep 29 03:18:10 dist-test-slave-c9qh krb5kdc[29344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.178.65@KRBTEST.COM for oryx/127.20.178.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:18:10 dist-test-slave-c9qh krb5kdc[29344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.178.65@KRBTEST.COM for oryx/127.20.178.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:18:10.239322 30077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:10.233943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:44981 (local address 127.20.178.65:36305)
0929 03:18:10.233996 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:10.234007 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 03:18:10.234046 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:18:10.234185 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:18:10.234186 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:18:10.234351 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:18:10.234406 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:10.234407 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:10.235275 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:10.235276 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:10.236174 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:10.236177 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:10.236695 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:10.236697 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:10.236770 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:10.237230 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:18:10.237235 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:18:10.239255 (+ 2020us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.178.126:44981: Server oryx/127.20.178.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":37}
W20260929 03:18:10.239566 30078 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:18:10.234032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.65:36305 (local address 127.20.178.126:44981)
0929 03:18:10.234065 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0929 03:18:10.234067 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:18:10.234075 (+ 8us) server_negotiation.cc:407] Connection header received
0929 03:18:10.234096 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:18:10.234096 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:18:10.234107 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:18:10.234170 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:18:10.234487 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:10.235189 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:10.236280 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:10.236586 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:10.236696 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:10.239507 (+ 2811us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.178.65:36305: BlockingRecv error: recv got EOF from 127.20.178.65:36305 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
Sep 29 03:18:10 dist-test-slave-c9qh krb5kdc[29344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.178.65@KRBTEST.COM for oryx/127.20.178.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:18:10 dist-test-slave-c9qh krb5kdc[29344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.178.65@KRBTEST.COM for oryx/127.20.178.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:18:10.244791 30077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:10.239584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:44981 (local address 127.20.178.65:57591)
0929 03:18:10.239629 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:10.239638 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 03:18:10.239673 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:18:10.240014 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:18:10.240014 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:18:10.240150 (+ 136us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:18:10.240179 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:10.240180 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:10.241083 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:10.241084 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:10.241882 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:10.241884 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:10.242339 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:10.242340 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:10.242391 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:10.242882 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:18:10.242886 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:18:10.244725 (+ 1839us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.178.126:44981: Server oryx/127.20.178.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":29}
W20260929 03:18:10.244927 30075 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 03:18:10.245048 30078 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:18:10.239640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.65:57591 (local address 127.20.178.126:44981)
0929 03:18:10.239877 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0929 03:18:10.239879 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:18:10.239889 (+ 10us) server_negotiation.cc:407] Connection header received
0929 03:18:10.239911 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:18:10.239912 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:18:10.239924 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:18:10.239991 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:18:10.240265 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:10.241001 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:10.241972 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:10.242263 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:10.242331 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:10.244812 (+ 2481us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.178.65:57591: BlockingRecv error: recv got EOF from 127.20.178.65:57591 (error 108)
Metrics: {"server-negotiator.queue_time_us":219,"thread_start_us":80,"threads_started":1}
W20260929 03:18:10.310467 30079 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:18:10.310693 30079 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:18:10.310740 30079 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:18:10.311875 30079 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:18:10.311941 30079 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:18:10.311985 30079 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.178.66
I20260929 03:18:10.313447 30079 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.178.84:44235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651852476208-21193-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651852476208-21193-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651852476208-21193-0/krb5kdc/oryx_127.20.178.66.keytab
--principal=oryx/127.20.178.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.178.66:33319
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651852476208-21193-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.178.66
--webserver_port=40973
--tserver_master_addrs=127.20.178.126:44981
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30079
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.178.66
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651852476208-21193-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:18:10.313722 30079 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:18:10.314013 30079 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:18:10.318854 30086 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:18:10.318871 30088 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:18:10.319010 30079 server_base.cc:1034] running on GCE node
W20260929 03:18:10.318882 30085 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:18:10.319293 30079 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:18:10.319515 30079 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:18:10.320693 30079 hybrid_clock.cc:648] HybridClock initialized: now 1790651890320704 us; error 69 us; skew 500 ppm
Sep 29 03:18:10 dist-test-slave-c9qh krb5kdc[29344](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651890, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.178.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:18:10.323215 30079 init.cc:377] Logged in from keytab as oryx/127.20.178.66@KRBTEST.COM (short username oryx)
I20260929 03:18:10.323670 30079 webserver.cc:460] Webserver started at http://127.20.178.66:40973/ using document root <none> and password file <none>
I20260929 03:18:10.323899 30079 fs_manager.cc:359] Metadata directory not provided
I20260929 03:18:10.323979 30079 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:18:10.325331 30079 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 03:18:10.325925 30095 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:18:10.326109 30079 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 03:18:10.326198 30079 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651852476208-21193-0/minicluster-data/ts-1/data,/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651852476208-21193-0/minicluster-data/ts-1/wal
uuid: "4cd1656091f14a86919ae2ed3063dc00"
format_stamp: "Formatted at 2026-09-29 03:18:09 on dist-test-slave-c9qh"
server_key: "259a53b0c8a46ef3e3aa98c09997401a"
server_key_iv: "f5f09ff3f118aabc0b12d7160f84617f"
server_key_version: "encryptionkey@0"
I20260929 03:18:10.326531 30079 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651852476208-21193-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651852476208-21193-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651852476208-21193-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:18:10.337589 30079 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:18:10.339270 30079 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:18:10.339422 30079 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:18:10.339623 30079 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:18:10.339912 30079 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:18:10.339965 30079 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:18:10.340004 30079 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:18:10.340032 30079 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:18:10.345990 30079 rpc_server.cc:307] RPC server started. Bound to: 127.20.178.66:33319
I20260929 03:18:10.346035 30208 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.178.66:33319 every 8 connection(s)
I20260929 03:18:10.346349 30079 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651852476208-21193-0/minicluster-data/ts-1/data/info.pb
I20260929 03:18:10.351729 21193 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuoaHbX/build/release/bin/kudu as pid 30079
I20260929 03:18:10.352013 21193 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuoaHbX/build/release/bin/kudu
/tmp/dist-test-taskuoaHbX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651852476208-21193-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651852476208-21193-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651852476208-21193-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651852476208-21193-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.178.67:44599
--local_ip_for_outbound_sockets=127.20.178.67
--tserver_master_addrs=127.20.178.126:44981
--webserver_port=39549
--webserver_interface=127.20.178.67
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.67.keytab
--principal=kudu/127.20.178.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.178.84:44235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.20.178.67
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651852476208-21193-0/krb5kdc/oryx_127.20.178.67.keytab with env {KRB5CCNAME=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651852476208-21193-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651852476208-21193-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651852476208-21193-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 03:18:10 dist-test-slave-c9qh krb5kdc[29344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.178.66@KRBTEST.COM for oryx/127.20.178.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:18:10 dist-test-slave-c9qh krb5kdc[29344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.178.66@KRBTEST.COM for oryx/127.20.178.126@KRBTEST.COM, Server not found in Kerberos database
W20260929 03:18:10.353916 30078 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:18:10.347009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.66:41393 (local address 127.20.178.126:44981)
0929 03:18:10.347090 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0929 03:18:10.347093 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:18:10.347294 (+ 201us) server_negotiation.cc:407] Connection header received
0929 03:18:10.347797 (+ 503us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:18:10.347798 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:18:10.347811 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:18:10.347864 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:18:10.348766 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:10.349243 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:10.350355 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:10.350525 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:10.350572 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:10.353870 (+ 3298us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.178.66:41393: BlockingRecv error: recv got EOF from 127.20.178.66:41393 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
I20260929 03:18:10.354012 30211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:10.347006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:44981 (local address 127.20.178.66:41393)
0929 03:18:10.347199 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:10.347232 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0929 03:18:10.347712 (+ 480us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:18:10.347933 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:18:10.347937 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:18:10.348474 (+ 537us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:18:10.348678 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:10.348682 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:10.349324 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:10.349325 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:10.350275 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:10.350278 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:10.350598 (+ 320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:10.350599 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:10.350728 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:10.351182 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:18:10.351191 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:18:10.353794 (+ 2603us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.178.126:44981: Server oryx/127.20.178.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":115,"threads_started":1}
W20260929 03:18:10.354249 30209 heartbeater.cc:644] Failed to heartbeat to 127.20.178.126:44981 (0 consecutive failures): Not authorized: Failed to ping master at 127.20.178.126:44981: Client connection negotiation failed: client connection to 127.20.178.126:44981: Server oryx/127.20.178.126@KRBTEST.COM not found in Kerberos database
Sep 29 03:18:10 dist-test-slave-c9qh krb5kdc[29344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.178.66@KRBTEST.COM for oryx/127.20.178.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:18:10 dist-test-slave-c9qh krb5kdc[29344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.178.66@KRBTEST.COM for oryx/127.20.178.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:18:10.359091 30211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:10.354470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:44981 (local address 127.20.178.66:41403)
0929 03:18:10.354511 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:10.354521 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 03:18:10.354554 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:18:10.354673 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:18:10.354674 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:18:10.354818 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:18:10.354869 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:10.354871 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:10.355451 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:10.355452 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:10.356348 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:10.356351 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:10.356701 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:10.356702 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:10.356776 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:10.357132 (+ 356us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:18:10.357138 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:18:10.359016 (+ 1878us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.178.126:44981: Server oryx/127.20.178.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":26}
W20260929 03:18:10.359144 30078 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:18:10.354523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.66:41403 (local address 127.20.178.126:44981)
0929 03:18:10.354560 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0929 03:18:10.354562 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:18:10.354572 (+ 10us) server_negotiation.cc:407] Connection header received
0929 03:18:10.354596 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:18:10.354597 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:18:10.354607 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:18:10.354653 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:18:10.354935 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:10.355363 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:10.356430 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:10.356595 (+ 165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:10.356697 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:10.359081 (+ 2384us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.178.66:41403: BlockingRecv error: recv got EOF from 127.20.178.66:41403 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
Sep 29 03:18:10 dist-test-slave-c9qh krb5kdc[29344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.178.66@KRBTEST.COM for oryx/127.20.178.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:18:10 dist-test-slave-c9qh krb5kdc[29344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.178.66@KRBTEST.COM for oryx/127.20.178.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:18:10.363528 30211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:10.359458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:44981 (local address 127.20.178.66:46449)
0929 03:18:10.359511 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:10.359521 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 03:18:10.359561 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:18:10.359715 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:18:10.359717 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:18:10.359838 (+ 121us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:18:10.359861 (+ 23us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:10.359862 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:10.360444 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:10.360445 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:10.360998 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:10.361000 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:10.361364 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:10.361365 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:10.361417 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:10.361797 (+ 380us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:18:10.361802 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:18:10.363458 (+ 1656us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.178.126:44981: Server oryx/127.20.178.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":37}
W20260929 03:18:10.363592 30078 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:18:10.359527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.66:46449 (local address 127.20.178.126:44981)
0929 03:18:10.359563 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0929 03:18:10.359565 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:18:10.359573 (+ 8us) server_negotiation.cc:407] Connection header received
0929 03:18:10.359615 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:18:10.359616 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:18:10.359624 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:18:10.359697 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:18:10.359940 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:10.360369 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:10.361086 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:10.361266 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:10.361365 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:10.363517 (+ 2152us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.178.66:46449: BlockingRecv error: recv got EOF from 127.20.178.66:46449 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260929 03:18:10.363658 30209 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 03:18:10.434767 30212 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:18:10.435025 30212 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:18:10.435062 30212 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:18:10.436244 30212 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:18:10.436322 30212 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:18:10.436368 30212 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.178.67
I20260929 03:18:10.437865 30212 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.178.84:44235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651852476208-21193-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651852476208-21193-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651852476208-21193-0/krb5kdc/oryx_127.20.178.67.keytab
--principal=oryx/127.20.178.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.178.67:44599
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651852476208-21193-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.178.67
--webserver_port=39549
--tserver_master_addrs=127.20.178.126:44981
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30212
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.178.67
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651852476208-21193-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:18:10.438139 30212 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:18:10.438367 30212 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:18:10.443368 30218 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:18:10.443519 30212 server_base.cc:1034] running on GCE node
W20260929 03:18:10.443323 30220 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:18:10.443388 30217 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:18:10.443845 30212 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:18:10.444108 30212 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:18:10.445266 30212 hybrid_clock.cc:648] HybridClock initialized: now 1790651890445262 us; error 34 us; skew 500 ppm
Sep 29 03:18:10 dist-test-slave-c9qh krb5kdc[29344](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651890, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.178.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:18:10.447770 30212 init.cc:377] Logged in from keytab as oryx/127.20.178.67@KRBTEST.COM (short username oryx)
I20260929 03:18:10.448180 30212 webserver.cc:460] Webserver started at http://127.20.178.67:39549/ using document root <none> and password file <none>
I20260929 03:18:10.448405 30212 fs_manager.cc:359] Metadata directory not provided
I20260929 03:18:10.448484 30212 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:18:10.449911 30212 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 03:18:10.450487 30227 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:18:10.450651 30212 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 03:18:10.450752 30212 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651852476208-21193-0/minicluster-data/ts-2/data,/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651852476208-21193-0/minicluster-data/ts-2/wal
uuid: "864babb0800d4871991b59bc820f2a7b"
format_stamp: "Formatted at 2026-09-29 03:18:09 on dist-test-slave-c9qh"
server_key: "9a8f92c62f2cf9b3024789457085de17"
server_key_iv: "0ec2eb0c16514ec191d88996c22f5bd9"
server_key_version: "encryptionkey@0"
I20260929 03:18:10.451097 30212 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651852476208-21193-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651852476208-21193-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651852476208-21193-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:18:10.461382 30212 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:18:10.463619 30212 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:18:10.463783 30212 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:18:10.464010 30212 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:18:10.464324 30212 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:18:10.464377 30212 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:18:10.464419 30212 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:18:10.464447 30212 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:18:10.470252 30212 rpc_server.cc:307] RPC server started. Bound to: 127.20.178.67:44599
I20260929 03:18:10.470290 30340 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.178.67:44599 every 8 connection(s)
I20260929 03:18:10.470650 30212 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651852476208-21193-0/minicluster-data/ts-2/data/info.pb
Sep 29 03:18:10 dist-test-slave-c9qh krb5kdc[29344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.178.67@KRBTEST.COM for oryx/127.20.178.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:18:10 dist-test-slave-c9qh krb5kdc[29344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.178.67@KRBTEST.COM for oryx/127.20.178.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:18:10.477723 21193 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuoaHbX/build/release/bin/kudu as pid 30212
W20260929 03:18:10.478190 30078 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:18:10.471473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.67:42729 (local address 127.20.178.126:44981)
0929 03:18:10.471542 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0929 03:18:10.471544 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:18:10.471731 (+ 187us) server_negotiation.cc:407] Connection header received
0929 03:18:10.472218 (+ 487us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:18:10.472219 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:18:10.472230 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:18:10.472290 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:18:10.473222 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:10.473704 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:10.474769 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:10.474961 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:10.475018 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:10.478102 (+ 3084us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.178.67:42729: BlockingRecv error: recv got EOF from 127.20.178.67:42729 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
I20260929 03:18:10.478229 30343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:10.471466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:44981 (local address 127.20.178.67:42729)
0929 03:18:10.471612 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:10.471646 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0929 03:18:10.472114 (+ 468us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:18:10.472339 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:18:10.472342 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:18:10.472934 (+ 592us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:18:10.473122 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:10.473127 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:10.473823 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:10.473824 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:10.474623 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:10.474625 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:10.475038 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:10.475039 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:10.475140 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:10.475732 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:18:10.475740 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:18:10.478006 (+ 2266us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.178.126:44981: Server oryx/127.20.178.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":86,"threads_started":1}
W20260929 03:18:10.478448 30341 heartbeater.cc:644] Failed to heartbeat to 127.20.178.126:44981 (0 consecutive failures): Not authorized: Failed to ping master at 127.20.178.126:44981: Client connection negotiation failed: client connection to 127.20.178.126:44981: Server oryx/127.20.178.126@KRBTEST.COM not found in Kerberos database
Sep 29 03:18:10 dist-test-slave-c9qh krb5kdc[29344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.178.67@KRBTEST.COM for oryx/127.20.178.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:18:10 dist-test-slave-c9qh krb5kdc[29344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.178.67@KRBTEST.COM for oryx/127.20.178.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:18:10.483779 30343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:10.478671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:44981 (local address 127.20.178.67:35329)
0929 03:18:10.478801 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:10.478809 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0929 03:18:10.478873 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:18:10.479031 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:18:10.479032 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:18:10.479163 (+ 131us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:18:10.479207 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:10.479208 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:10.479870 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:10.479871 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:10.480374 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:10.480375 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:10.480833 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:10.480833 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:10.480896 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:10.481366 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:18:10.481371 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:18:10.483732 (+ 2361us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.178.126:44981: Server oryx/127.20.178.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":117}
W20260929 03:18:10.483946 30080 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:18:10.478716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.67:35329 (local address 127.20.178.126:44981)
0929 03:18:10.478760 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0929 03:18:10.478761 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 03:18:10.478862 (+ 101us) server_negotiation.cc:407] Connection header received
0929 03:18:10.478930 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:18:10.478931 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:18:10.478941 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:18:10.479028 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:18:10.479294 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:10.479792 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:10.480471 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:10.480750 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:10.480833 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:10.483872 (+ 3039us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.178.67:35329: BlockingRecv error: recv got EOF from 127.20.178.67:35329 (error 108)
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":18}
I20260929 03:18:10.486261 30078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:10.478252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53104 (local address 127.20.178.126:44981)
0929 03:18:10.478303 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0929 03:18:10.478305 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:18:10.478503 (+ 198us) server_negotiation.cc:407] Connection header received
0929 03:18:10.478618 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:18:10.478619 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:18:10.478628 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:18:10.478710 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:18:10.479318 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:10.479789 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:10.480529 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:10.480790 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:10.480861 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:10.482221 (+ 1360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:18:10.482226 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:18:10.482226 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:18:10.482238 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:18:10.484826 (+ 2588us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:18:10.485158 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:18:10.485161 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:18:10.485162 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:18:10.485282 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:18:10.485491 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:18:10.485491 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:18:10.485492 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:18:10.485801 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:18:10.485914 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0929 03:18:10.486082 (+ 168us) server_negotiation.cc:299] Negotiation successful
0929 03:18:10.486165 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
Sep 29 03:18:10 dist-test-slave-c9qh krb5kdc[29344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.178.67@KRBTEST.COM for oryx/127.20.178.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:18:10 dist-test-slave-c9qh krb5kdc[29344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.178.67@KRBTEST.COM for oryx/127.20.178.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:18:10.488912 30343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:10.484195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:44981 (local address 127.20.178.67:39307)
0929 03:18:10.484245 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:10.484255 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 03:18:10.484296 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:18:10.484520 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:18:10.484521 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:18:10.484720 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:18:10.484756 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:10.484757 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:10.485608 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:10.485609 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:10.486340 (+ 731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:10.486343 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:10.486814 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:10.486815 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:10.486851 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:10.487302 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:18:10.487307 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:18:10.488856 (+ 1549us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.178.126:44981: Server oryx/127.20.178.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":34}
W20260929 03:18:10.489035 30080 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:18:10.484255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.67:39307 (local address 127.20.178.126:44981)
0929 03:18:10.484289 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0929 03:18:10.484290 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 03:18:10.484299 (+ 9us) server_negotiation.cc:407] Connection header received
0929 03:18:10.484352 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:18:10.484353 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:18:10.484363 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:18:10.484416 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:18:10.484830 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:10.485518 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:10.486445 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:10.486703 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:10.486746 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:10.488962 (+ 2216us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.178.67:39307: BlockingRecv error: recv got EOF from 127.20.178.67:39307 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260929 03:18:10.489037 30341 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 29 03:18:11 dist-test-slave-c9qh krb5kdc[29344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.178.65@KRBTEST.COM for oryx/127.20.178.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:18:11 dist-test-slave-c9qh krb5kdc[29344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.178.65@KRBTEST.COM for oryx/127.20.178.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:18:11.250244 30345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:11.245321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:44981 (local address 127.20.178.65:60453)
0929 03:18:11.245460 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:11.245471 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 03:18:11.245532 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:18:11.245725 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:18:11.245726 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:18:11.245883 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:18:11.245933 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:11.245935 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:11.246693 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:11.246695 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:11.247519 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:11.247521 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:11.247874 (+ 353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:11.247874 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:11.247914 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:11.248344 (+ 430us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:18:11.248348 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:18:11.250167 (+ 1819us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.178.126:44981: Server oryx/127.20.178.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":90,"threads_started":1}
W20260929 03:18:11.250306 30346 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:18:11.245421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.65:60453 (local address 127.20.178.126:44981)
0929 03:18:11.245559 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0929 03:18:11.245562 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:18:11.245570 (+ 8us) server_negotiation.cc:407] Connection header received
0929 03:18:11.245598 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:18:11.245599 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:18:11.245611 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:18:11.245685 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:18:11.246043 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:11.246545 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:11.247614 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:11.247808 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:11.247862 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:11.250253 (+ 2391us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.178.65:60453: BlockingRecv error: recv got EOF from 127.20.178.65:60453 (error 108)
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":86,"threads_started":1}
Sep 29 03:18:11 dist-test-slave-c9qh krb5kdc[29344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.178.66@KRBTEST.COM for oryx/127.20.178.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:18:11 dist-test-slave-c9qh krb5kdc[29344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.178.66@KRBTEST.COM for oryx/127.20.178.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:18:11.369202 30347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:11.364073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:44981 (local address 127.20.178.66:37583)
0929 03:18:11.364248 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:11.364257 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 03:18:11.364339 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:18:11.364485 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:18:11.364486 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:18:11.364713 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:18:11.364771 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:11.364774 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:11.365484 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:11.365485 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:11.366215 (+ 730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:11.366218 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:11.366692 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:11.366693 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:11.366741 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:11.367168 (+ 427us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:18:11.367172 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:18:11.369128 (+ 1956us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.178.126:44981: Server oryx/127.20.178.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":67,"threads_started":1}
W20260929 03:18:11.369256 30346 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:18:11.364128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.66:37583 (local address 127.20.178.126:44981)
0929 03:18:11.364213 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0929 03:18:11.364215 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:18:11.364319 (+ 104us) server_negotiation.cc:407] Connection header received
0929 03:18:11.364423 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:18:11.364424 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:18:11.364435 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:18:11.364481 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:18:11.364853 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:11.365405 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:11.366301 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:11.366618 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:11.366674 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:11.369207 (+ 2533us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.178.66:37583: BlockingRecv error: recv got EOF from 127.20.178.66:37583 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 29 03:18:11 dist-test-slave-c9qh krb5kdc[29344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.178.67@KRBTEST.COM for oryx/127.20.178.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:18:11 dist-test-slave-c9qh krb5kdc[29344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.178.67@KRBTEST.COM for oryx/127.20.178.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:18:11.493981 30348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:11.489419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:44981 (local address 127.20.178.67:39517)
0929 03:18:11.489584 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:11.489595 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 03:18:11.489682 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:18:11.489807 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:18:11.489808 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:18:11.489987 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:18:11.490033 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:11.490035 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:11.490697 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:11.490698 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:11.491246 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:11.491248 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:11.491620 (+ 372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:11.491620 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:11.491685 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:11.492104 (+ 419us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:18:11.492108 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:18:11.493917 (+ 1809us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.178.126:44981: Server oryx/127.20.178.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":82,"threads_started":1}
W20260929 03:18:11.494107 30346 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:18:11.489503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.67:39517 (local address 127.20.178.126:44981)
0929 03:18:11.489576 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0929 03:18:11.489578 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:18:11.489662 (+ 84us) server_negotiation.cc:407] Connection header received
0929 03:18:11.489737 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:18:11.489738 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:18:11.489752 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:18:11.489803 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:18:11.490112 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:11.490602 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:11.491333 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:11.491532 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:11.491618 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:11.494007 (+ 2389us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.178.67:39517: BlockingRecv error: recv got EOF from 127.20.178.67:39517 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 29 03:18:12 dist-test-slave-c9qh krb5kdc[29344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.178.65@KRBTEST.COM for oryx/127.20.178.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:18:12 dist-test-slave-c9qh krb5kdc[29344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.178.65@KRBTEST.COM for oryx/127.20.178.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:18:12.255429 30349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:12.250771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:44981 (local address 127.20.178.65:60227)
0929 03:18:12.250913 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:12.250924 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 03:18:12.251011 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:18:12.251219 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:18:12.251220 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:18:12.251424 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:18:12.251470 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:12.251472 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:12.252091 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:12.252091 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:12.252703 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:12.252705 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:12.253088 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:12.253088 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:12.253135 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:12.253672 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:18:12.253677 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:18:12.255370 (+ 1693us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.178.126:44981: Server oryx/127.20.178.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":61,"threads_started":1}
W20260929 03:18:12.255532 30350 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:18:12.250826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.65:60227 (local address 127.20.178.126:44981)
0929 03:18:12.250962 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0929 03:18:12.250964 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:18:12.250996 (+ 32us) server_negotiation.cc:407] Connection header received
0929 03:18:12.251079 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:18:12.251080 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:18:12.251092 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:18:12.251159 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:18:12.251536 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:12.252021 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:12.252787 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:12.253017 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:12.253109 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:12.255458 (+ 2349us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.178.65:60227: BlockingRecv error: recv got EOF from 127.20.178.65:60227 (error 108)
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":80,"threads_started":1}
Sep 29 03:18:12 dist-test-slave-c9qh krb5kdc[29344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.178.66@KRBTEST.COM for oryx/127.20.178.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:18:12 dist-test-slave-c9qh krb5kdc[29344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.178.66@KRBTEST.COM for oryx/127.20.178.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:18:12.374356 30351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:12.369702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:44981 (local address 127.20.178.66:40605)
0929 03:18:12.369815 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:12.369833 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 03:18:12.369913 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:18:12.370081 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:18:12.370082 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:18:12.370273 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:18:12.370322 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:12.370324 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:12.371032 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:12.371033 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:12.371562 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:12.371564 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:12.371952 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:12.371953 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:12.372017 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:12.372454 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:18:12.372459 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:18:12.374299 (+ 1840us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.178.126:44981: Server oryx/127.20.178.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":64,"threads_started":1}
W20260929 03:18:12.374460 30350 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:18:12.369713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.66:40605 (local address 127.20.178.126:44981)
0929 03:18:12.369784 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0929 03:18:12.369786 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:18:12.369902 (+ 116us) server_negotiation.cc:407] Connection header received
0929 03:18:12.369971 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:18:12.369972 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:18:12.369986 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:18:12.370076 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:18:12.370408 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:12.370926 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:12.371669 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:12.371867 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:12.371944 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:12.374392 (+ 2448us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.178.66:40605: BlockingRecv error: recv got EOF from 127.20.178.66:40605 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 29 03:18:12 dist-test-slave-c9qh krb5kdc[29344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.178.67@KRBTEST.COM for oryx/127.20.178.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:18:12 dist-test-slave-c9qh krb5kdc[29344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.178.67@KRBTEST.COM for oryx/127.20.178.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:18:12.499018 30352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:12.494381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:44981 (local address 127.20.178.67:56053)
0929 03:18:12.494514 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:12.494544 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0929 03:18:12.494602 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:18:12.494776 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:18:12.494777 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:18:12.494948 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:18:12.494995 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:12.494997 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:12.495664 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:12.495665 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:12.496199 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:12.496200 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:12.496550 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:12.496551 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:12.496617 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:12.497141 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:18:12.497146 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:18:12.498921 (+ 1775us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.178.126:44981: Server oryx/127.20.178.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":74,"threads_started":1}
W20260929 03:18:12.499101 30350 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:18:12.494502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.67:56053 (local address 127.20.178.126:44981)
0929 03:18:12.494572 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0929 03:18:12.494574 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:18:12.494584 (+ 10us) server_negotiation.cc:407] Connection header received
0929 03:18:12.494686 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:18:12.494687 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:18:12.494708 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:18:12.494769 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:18:12.495085 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:12.495578 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:12.496287 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:12.496476 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:12.496552 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:12.499009 (+ 2457us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.178.67:56053: BlockingRecv error: recv got EOF from 127.20.178.67:56053 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 29 03:18:13 dist-test-slave-c9qh krb5kdc[29344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.178.65@KRBTEST.COM for oryx/127.20.178.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:18:13 dist-test-slave-c9qh krb5kdc[29344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.178.65@KRBTEST.COM for oryx/127.20.178.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:18:13.260617 30353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:13.255979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:44981 (local address 127.20.178.65:44539)
0929 03:18:13.256142 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:13.256162 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 03:18:13.256227 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:18:13.256374 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:18:13.256375 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:18:13.256528 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:18:13.256576 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:13.256578 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:13.257250 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:13.257250 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:13.257995 (+ 745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:13.257998 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:13.258323 (+ 325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:13.258324 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:13.258362 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:13.258841 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:18:13.258846 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:18:13.260559 (+ 1713us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.178.126:44981: Server oryx/127.20.178.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":80,"threads_started":1}
W20260929 03:18:13.260778 30354 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:18:13.256033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.65:44539 (local address 127.20.178.126:44981)
0929 03:18:13.256189 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0929 03:18:13.256191 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:18:13.256201 (+ 10us) server_negotiation.cc:407] Connection header received
0929 03:18:13.256284 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:18:13.256285 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:18:13.256295 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:18:13.256354 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:18:13.256690 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:13.257181 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:13.258068 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:13.258264 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:13.258316 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:13.260688 (+ 2372us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.178.65:44539: BlockingRecv error: recv got EOF from 127.20.178.65:44539 (error 108)
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":80,"threads_started":1}
Sep 29 03:18:13 dist-test-slave-c9qh krb5kdc[29344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.178.66@KRBTEST.COM for oryx/127.20.178.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:18:13 dist-test-slave-c9qh krb5kdc[29344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.178.66@KRBTEST.COM for oryx/127.20.178.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:18:13.379410 30355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:13.374828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:44981 (local address 127.20.178.66:56267)
0929 03:18:13.374984 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:13.374994 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 03:18:13.375050 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:18:13.375183 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:18:13.375184 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:18:13.375338 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:18:13.375385 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:13.375387 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:13.376202 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:13.376203 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:13.376896 (+ 693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:13.376898 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:13.377231 (+ 333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:13.377231 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:13.377268 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:13.377683 (+ 415us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:18:13.377687 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:18:13.379355 (+ 1668us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.178.126:44981: Server oryx/127.20.178.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":89,"threads_started":1}
W20260929 03:18:13.379504 30354 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:18:13.374903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.66:56267 (local address 127.20.178.126:44981)
0929 03:18:13.374975 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0929 03:18:13.374976 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 03:18:13.375035 (+ 59us) server_negotiation.cc:407] Connection header received
0929 03:18:13.375108 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:18:13.375108 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:18:13.375118 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:18:13.375159 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:18:13.375449 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:13.376120 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:13.376970 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:13.377162 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:13.377217 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:13.379447 (+ 2230us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.178.66:56267: BlockingRecv error: recv got EOF from 127.20.178.66:56267 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Sep 29 03:18:13 dist-test-slave-c9qh krb5kdc[29344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.178.67@KRBTEST.COM for oryx/127.20.178.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:18:13 dist-test-slave-c9qh krb5kdc[29344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.178.67@KRBTEST.COM for oryx/127.20.178.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:18:13.504364 30356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:13.499694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:44981 (local address 127.20.178.67:53573)
0929 03:18:13.499811 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:13.499821 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 03:18:13.499867 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:18:13.500017 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:18:13.500018 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:18:13.500175 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:18:13.500223 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:13.500225 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:13.500939 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:13.500939 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:13.501544 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:13.501547 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:13.501907 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:13.501908 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:13.501949 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:13.502505 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:18:13.502510 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:18:13.504271 (+ 1761us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.178.126:44981: Server oryx/127.20.178.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":64,"threads_started":1}
W20260929 03:18:13.504403 30354 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:18:13.499780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.67:53573 (local address 127.20.178.126:44981)
0929 03:18:13.499853 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0929 03:18:13.499855 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:18:13.499865 (+ 10us) server_negotiation.cc:407] Connection header received
0929 03:18:13.499927 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:18:13.499928 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:18:13.499939 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:18:13.499998 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:18:13.500292 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:13.500876 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:13.501657 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:13.501835 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:13.501895 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:13.504343 (+ 2448us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.178.67:53573: BlockingRecv error: recv got EOF from 127.20.178.67:53573 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 29 03:18:14 dist-test-slave-c9qh krb5kdc[29344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.178.65@KRBTEST.COM for oryx/127.20.178.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:18:14 dist-test-slave-c9qh krb5kdc[29344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.178.65@KRBTEST.COM for oryx/127.20.178.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:18:14.265753 30357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:14.261099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:44981 (local address 127.20.178.65:46205)
0929 03:18:14.261253 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:14.261263 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 03:18:14.261312 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:18:14.261454 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:18:14.261454 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:18:14.261611 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:18:14.261668 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:14.261670 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:14.262295 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:14.262295 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:14.262829 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:14.262831 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:14.263149 (+ 318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:14.263150 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:14.263209 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:14.263711 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:18:14.263716 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:18:14.265694 (+ 1978us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.178.126:44981: Server oryx/127.20.178.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":86,"threads_started":1}
W20260929 03:18:14.265861 30358 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:18:14.261164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.65:46205 (local address 127.20.178.126:44981)
0929 03:18:14.261292 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0929 03:18:14.261294 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:18:14.261303 (+ 9us) server_negotiation.cc:407] Connection header received
0929 03:18:14.261369 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:18:14.261369 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:18:14.261379 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:18:14.261438 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:18:14.261739 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:14.262224 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:14.262899 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:14.263083 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:14.263131 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:14.265786 (+ 2655us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.178.65:46205: BlockingRecv error: recv got EOF from 127.20.178.65:46205 (error 108)
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":61,"threads_started":1}
Sep 29 03:18:14 dist-test-slave-c9qh krb5kdc[29344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.178.66@KRBTEST.COM for oryx/127.20.178.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:18:14 dist-test-slave-c9qh krb5kdc[29344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.178.66@KRBTEST.COM for oryx/127.20.178.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:18:14.389743 30359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:14.380086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:44981 (local address 127.20.178.66:39455)
0929 03:18:14.380267 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:14.380280 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 03:18:14.380381 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:18:14.380519 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:18:14.380520 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:18:14.380723 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:18:14.380770 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:14.380772 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:14.381465 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:14.381466 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:14.382013 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:14.382014 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:14.382399 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:14.382400 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:14.382459 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:14.387832 (+ 5373us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:18:14.387837 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:18:14.389677 (+ 1840us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.178.126:44981: Server oryx/127.20.178.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":89,"threads_started":1}
W20260929 03:18:14.389860 30358 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:18:14.380185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.66:39455 (local address 127.20.178.126:44981)
0929 03:18:14.380270 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0929 03:18:14.380272 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:18:14.380361 (+ 89us) server_negotiation.cc:407] Connection header received
0929 03:18:14.380449 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:18:14.380450 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:18:14.380463 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:18:14.380516 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:18:14.380854 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:14.381373 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:14.382103 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:14.382307 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:14.382398 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:14.389782 (+ 7384us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.178.66:39455: BlockingRecv error: recv got EOF from 127.20.178.66:39455 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 29 03:18:14 dist-test-slave-c9qh krb5kdc[29344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.178.67@KRBTEST.COM for oryx/127.20.178.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:18:14 dist-test-slave-c9qh krb5kdc[29344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.178.67@KRBTEST.COM for oryx/127.20.178.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:18:14.509509 30360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:14.504864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:44981 (local address 127.20.178.67:34429)
0929 03:18:14.504995 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:14.505012 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 03:18:14.505069 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:18:14.505207 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:18:14.505207 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:18:14.505364 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:18:14.505411 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:14.505413 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:14.506079 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:14.506080 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:14.506598 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:14.506600 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:14.506998 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:14.506999 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:14.507040 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:14.507503 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:18:14.507507 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:18:14.509413 (+ 1906us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.178.126:44981: Server oryx/127.20.178.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":81,"threads_started":1}
W20260929 03:18:14.509558 30358 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:18:14.504935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.67:34429 (local address 127.20.178.126:44981)
0929 03:18:14.505012 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0929 03:18:14.505014 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:18:14.505070 (+ 56us) server_negotiation.cc:407] Connection header received
0929 03:18:14.505130 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:18:14.505130 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:18:14.505144 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:18:14.505186 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:18:14.505475 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:14.505972 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:14.506739 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:14.506926 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:14.506991 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:14.509492 (+ 2501us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.178.67:34429: BlockingRecv error: recv got EOF from 127.20.178.67:34429 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 29 03:18:15 dist-test-slave-c9qh krb5kdc[29344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.178.65@KRBTEST.COM for oryx/127.20.178.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:18:15 dist-test-slave-c9qh krb5kdc[29344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.178.65@KRBTEST.COM for oryx/127.20.178.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:18:15.270985 30361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:15.266373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:44981 (local address 127.20.178.65:55493)
0929 03:18:15.266529 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:15.266539 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 03:18:15.266719 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:18:15.266859 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:18:15.266860 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:18:15.267047 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:18:15.267097 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:15.267098 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:15.267795 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:15.267796 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:15.268317 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:15.268319 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:15.268724 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:15.268725 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:15.268788 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:15.269230 (+ 442us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:18:15.269234 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:18:15.270926 (+ 1692us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.178.126:44981: Server oryx/127.20.178.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":64,"threads_started":1}
W20260929 03:18:15.271090 30362 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:18:15.266437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.65:55493 (local address 127.20.178.126:44981)
0929 03:18:15.266546 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0929 03:18:15.266548 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:18:15.266707 (+ 159us) server_negotiation.cc:407] Connection header received
0929 03:18:15.266779 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:18:15.266779 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:18:15.266795 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:18:15.266854 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:18:15.267194 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:15.267698 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:15.268408 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:15.268615 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:15.268723 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:15.271014 (+ 2291us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.178.65:55493: BlockingRecv error: recv got EOF from 127.20.178.65:55493 (error 108)
Metrics: {"server-negotiator.queue_time_us":88,"thread_start_us":55,"threads_started":1}
Sep 29 03:18:15 dist-test-slave-c9qh krb5kdc[29344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.178.66@KRBTEST.COM for oryx/127.20.178.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:18:15 dist-test-slave-c9qh krb5kdc[29344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.178.66@KRBTEST.COM for oryx/127.20.178.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:18:15.395056 30363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:15.390263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:44981 (local address 127.20.178.66:57561)
0929 03:18:15.390459 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:15.390476 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 03:18:15.390537 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:18:15.390696 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:18:15.390697 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:18:15.390851 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:18:15.390899 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:15.390900 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:15.391644 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:15.391645 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:15.392276 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:15.392278 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:15.392605 (+ 327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:15.392606 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:15.392745 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:15.393178 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:18:15.393183 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:18:15.394964 (+ 1781us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.178.126:44981: Server oryx/127.20.178.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":108,"threads_started":1}
W20260929 03:18:15.395107 30362 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:18:15.390419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.66:57561 (local address 127.20.178.126:44981)
0929 03:18:15.390479 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0929 03:18:15.390481 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:18:15.390520 (+ 39us) server_negotiation.cc:407] Connection header received
0929 03:18:15.390615 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:18:15.390616 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:18:15.390627 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:18:15.390677 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:18:15.390990 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:15.391471 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:15.392343 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:15.392524 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:15.392589 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:15.395041 (+ 2452us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.178.66:57561: BlockingRecv error: recv got EOF from 127.20.178.66:57561 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Sep 29 03:18:15 dist-test-slave-c9qh krb5kdc[29344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.178.67@KRBTEST.COM for oryx/127.20.178.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:18:15 dist-test-slave-c9qh krb5kdc[29344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.178.67@KRBTEST.COM for oryx/127.20.178.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:18:15.514441 30364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:15.510056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:44981 (local address 127.20.178.67:59735)
0929 03:18:15.510208 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:15.510218 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 03:18:15.510278 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:18:15.510423 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:18:15.510424 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:18:15.510577 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:18:15.510624 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:15.510625 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:15.511240 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:15.511241 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:15.511774 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:15.511776 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:15.512076 (+ 300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:15.512076 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:15.512116 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:15.512541 (+ 425us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:18:15.512545 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:18:15.514384 (+ 1839us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.178.126:44981: Server oryx/127.20.178.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":77,"threads_started":1}
W20260929 03:18:15.514569 30362 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:18:15.510152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.67:59735 (local address 127.20.178.126:44981)
0929 03:18:15.510200 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0929 03:18:15.510202 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:18:15.510260 (+ 58us) server_negotiation.cc:407] Connection header received
0929 03:18:15.510338 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:18:15.510338 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:18:15.510350 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:18:15.510400 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:18:15.510691 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:15.511167 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:15.511845 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:15.512013 (+ 168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:15.512060 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:15.514468 (+ 2408us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.178.67:59735: BlockingRecv error: recv got EOF from 127.20.178.67:59735 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
Sep 29 03:18:16 dist-test-slave-c9qh krb5kdc[29344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.178.65@KRBTEST.COM for oryx/127.20.178.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:18:16 dist-test-slave-c9qh krb5kdc[29344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.178.65@KRBTEST.COM for oryx/127.20.178.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:18:16.276227 30365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:16.271495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:44981 (local address 127.20.178.65:47629)
0929 03:18:16.271660 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:16.271673 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 03:18:16.271762 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:18:16.271949 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:18:16.271949 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:18:16.272179 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:18:16.272242 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:16.272244 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:16.272922 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:16.272923 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:16.273439 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:16.273440 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:16.273799 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:16.273800 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:16.273839 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:16.274348 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:18:16.274353 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:18:16.276159 (+ 1806us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.178.126:44981: Server oryx/127.20.178.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":91,"threads_started":1}
W20260929 03:18:16.276378 30366 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:18:16.271581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.65:47629 (local address 127.20.178.126:44981)
0929 03:18:16.271676 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0929 03:18:16.271678 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:18:16.271743 (+ 65us) server_negotiation.cc:407] Connection header received
0929 03:18:16.271834 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:18:16.271835 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:18:16.271850 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:18:16.271947 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:18:16.272316 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:16.272858 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:16.273531 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:16.273730 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:16.273790 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:16.276287 (+ 2497us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.178.65:47629: BlockingRecv error: recv got EOF from 127.20.178.65:47629 (error 108)
Metrics: {"server-negotiator.queue_time_us":73,"thread_start_us":45,"threads_started":1}
Sep 29 03:18:16 dist-test-slave-c9qh krb5kdc[29344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.178.66@KRBTEST.COM for oryx/127.20.178.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:18:16 dist-test-slave-c9qh krb5kdc[29344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.178.66@KRBTEST.COM for oryx/127.20.178.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:18:16.402218 30367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:16.395600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:44981 (local address 127.20.178.66:46815)
0929 03:18:16.395775 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:16.395786 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 03:18:16.395843 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:18:16.395969 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:18:16.395970 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:18:16.396129 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:18:16.396200 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:16.396202 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:16.396915 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:16.396917 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:16.397483 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:16.397485 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:16.397927 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:16.397928 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:16.397993 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:16.398595 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:18:16.398599 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:18:16.402119 (+ 3520us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.178.126:44981: Server oryx/127.20.178.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":107,"threads_started":1}
W20260929 03:18:16.402433 30366 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:18:16.395666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.66:46815 (local address 127.20.178.126:44981)
0929 03:18:16.395757 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0929 03:18:16.395759 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:18:16.395844 (+ 85us) server_negotiation.cc:407] Connection header received
0929 03:18:16.395892 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:18:16.395892 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:18:16.395903 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:18:16.395951 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:18:16.396268 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:16.396828 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:16.397596 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:16.397821 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:16.397925 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:16.402219 (+ 4294us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.178.66:46815: BlockingRecv error: recv got EOF from 127.20.178.66:46815 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 29 03:18:16 dist-test-slave-c9qh krb5kdc[29344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.178.67@KRBTEST.COM for oryx/127.20.178.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:18:16 dist-test-slave-c9qh krb5kdc[29344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.178.67@KRBTEST.COM for oryx/127.20.178.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:18:16.519747 30368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:16.514866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:44981 (local address 127.20.178.67:51003)
0929 03:18:16.514983 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:16.514995 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 03:18:16.515064 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:18:16.515226 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:18:16.515227 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:18:16.515390 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:18:16.515434 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:16.515436 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:16.516150 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:16.516151 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:16.516771 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:16.516773 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:16.517216 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:16.517217 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:16.517276 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:16.517757 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:18:16.517763 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:18:16.519634 (+ 1871us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.178.126:44981: Server oryx/127.20.178.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":67,"threads_started":1}
W20260929 03:18:16.519778 30366 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:18:16.514914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.67:51003 (local address 127.20.178.126:44981)
0929 03:18:16.514964 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0929 03:18:16.514966 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:18:16.515048 (+ 82us) server_negotiation.cc:407] Connection header received
0929 03:18:16.515107 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:18:16.515107 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:18:16.515119 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:18:16.515184 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:18:16.515521 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:16.516070 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:16.516853 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:16.517119 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:16.517210 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:16.519720 (+ 2510us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.178.67:51003: BlockingRecv error: recv got EOF from 127.20.178.67:51003 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
Sep 29 03:18:17 dist-test-slave-c9qh krb5kdc[29344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.178.65@KRBTEST.COM for oryx/127.20.178.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:18:17 dist-test-slave-c9qh krb5kdc[29344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.178.65@KRBTEST.COM for oryx/127.20.178.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:18:17.281100 30369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:17.276667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:44981 (local address 127.20.178.65:45867)
0929 03:18:17.276822 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:17.276833 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 03:18:17.276886 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:18:17.277027 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:18:17.277028 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:18:17.277182 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:18:17.277244 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:17.277246 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:17.277899 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:17.277900 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:17.278416 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:17.278417 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:17.278808 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:17.278809 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:17.278848 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:17.279271 (+ 423us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:18:17.279275 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:18:17.281044 (+ 1769us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.178.126:44981: Server oryx/127.20.178.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":101,"threads_started":1}
W20260929 03:18:17.281246 30370 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:18:17.276781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.65:45867 (local address 127.20.178.126:44981)
0929 03:18:17.276901 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0929 03:18:17.276903 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:18:17.276912 (+ 9us) server_negotiation.cc:407] Connection header received
0929 03:18:17.276934 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:18:17.276934 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:18:17.276945 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:18:17.277013 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:18:17.277313 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:17.277834 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:17.278494 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:17.278718 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:17.278775 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:17.281144 (+ 2369us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.178.65:45867: BlockingRecv error: recv got EOF from 127.20.178.65:45867 (error 108)
Metrics: {"server-negotiator.queue_time_us":102,"thread_start_us":68,"threads_started":1}
Sep 29 03:18:17 dist-test-slave-c9qh krb5kdc[29344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.178.66@KRBTEST.COM for oryx/127.20.178.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:18:17 dist-test-slave-c9qh krb5kdc[29344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.178.66@KRBTEST.COM for oryx/127.20.178.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:18:17.407490 30371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:17.402927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:44981 (local address 127.20.178.66:40923)
0929 03:18:17.403094 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:17.403112 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 03:18:17.403195 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:18:17.403387 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:18:17.403388 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:18:17.403555 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:18:17.403600 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:17.403602 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:17.404240 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:17.404241 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:17.404835 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:17.404837 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:17.405219 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:17.405220 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:17.405273 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:17.405718 (+ 445us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:18:17.405722 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:18:17.407434 (+ 1712us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.178.126:44981: Server oryx/127.20.178.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":84,"threads_started":1}
W20260929 03:18:17.407647 30370 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:18:17.402987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.66:40923 (local address 127.20.178.126:44981)
0929 03:18:17.403056 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0929 03:18:17.403058 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:18:17.403177 (+ 119us) server_negotiation.cc:407] Connection header received
0929 03:18:17.403283 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:18:17.403284 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:18:17.403310 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:18:17.403380 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:18:17.403684 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:17.404162 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:17.404926 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:17.405121 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:17.405215 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:17.407537 (+ 2322us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.178.66:40923: BlockingRecv error: recv got EOF from 127.20.178.66:40923 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Sep 29 03:18:17 dist-test-slave-c9qh krb5kdc[29344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.178.67@KRBTEST.COM for oryx/127.20.178.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:18:17 dist-test-slave-c9qh krb5kdc[29344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.178.67@KRBTEST.COM for oryx/127.20.178.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:18:17.525683 30372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:17.520300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:44981 (local address 127.20.178.67:41223)
0929 03:18:17.520496 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:17.520508 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 03:18:17.520593 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:18:17.520786 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:18:17.520787 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:18:17.521015 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:18:17.521072 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:17.521075 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:17.521858 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:17.521860 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:17.522384 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:17.522386 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:17.522838 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:17.522839 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:17.522902 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:17.523432 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:18:17.523438 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:18:17.525579 (+ 2141us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.178.126:44981: Server oryx/127.20.178.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":99,"threads_started":1}
W20260929 03:18:17.525735 30370 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:18:17.520397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.67:41223 (local address 127.20.178.126:44981)
0929 03:18:17.520480 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0929 03:18:17.520482 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:18:17.520580 (+ 98us) server_negotiation.cc:407] Connection header received
0929 03:18:17.520689 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:18:17.520691 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:18:17.520705 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:18:17.520779 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:18:17.521170 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:17.521771 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:17.522502 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:17.522755 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:17.522840 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:17.525663 (+ 2823us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.178.67:41223: BlockingRecv error: recv got EOF from 127.20.178.67:41223 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 29 03:18:18 dist-test-slave-c9qh krb5kdc[29344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.178.65@KRBTEST.COM for oryx/127.20.178.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:18:18 dist-test-slave-c9qh krb5kdc[29344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.178.65@KRBTEST.COM for oryx/127.20.178.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:18:18.286237 30373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:18.281673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:44981 (local address 127.20.178.65:35447)
0929 03:18:18.281839 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:18.281854 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 03:18:18.281920 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:18:18.282106 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:18:18.282107 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:18:18.282262 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:18:18.282325 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:18.282327 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:18.282977 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:18.282978 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:18.283491 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:18.283492 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:18.283846 (+ 354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:18.283847 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:18.283885 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:18.284329 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:18:18.284333 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:18:18.286130 (+ 1797us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.178.126:44981: Server oryx/127.20.178.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":75,"threads_started":1}
W20260929 03:18:18.286284 30374 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:18:18.281821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.65:35447 (local address 127.20.178.126:44981)
0929 03:18:18.281983 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0929 03:18:18.281985 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:18:18.281992 (+ 7us) server_negotiation.cc:407] Connection header received
0929 03:18:18.282013 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:18:18.282013 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:18:18.282024 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:18:18.282087 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:18:18.282394 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:18.282913 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:18.283577 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:18.283781 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:18.283833 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:18.286213 (+ 2380us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.178.65:35447: BlockingRecv error: recv got EOF from 127.20.178.65:35447 (error 108)
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":83,"threads_started":1}
Sep 29 03:18:18 dist-test-slave-c9qh krb5kdc[29344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.178.66@KRBTEST.COM for oryx/127.20.178.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:18:18 dist-test-slave-c9qh krb5kdc[29344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.178.66@KRBTEST.COM for oryx/127.20.178.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:18:18.412312 30375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:18.407904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:44981 (local address 127.20.178.66:41007)
0929 03:18:18.408024 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:18.408035 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 03:18:18.408093 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:18:18.408238 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:18:18.408239 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:18:18.408397 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:18:18.408443 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:18.408444 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:18.409235 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:18.409235 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:18.409743 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:18.409744 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:18.410070 (+ 326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:18.410071 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:18.410110 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:18.410532 (+ 422us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:18:18.410536 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:18:18.412247 (+ 1711us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.178.126:44981: Server oryx/127.20.178.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":74,"threads_started":1}
W20260929 03:18:18.412410 30374 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:18:18.407995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.66:41007 (local address 127.20.178.126:44981)
0929 03:18:18.408061 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0929 03:18:18.408063 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:18:18.408079 (+ 16us) server_negotiation.cc:407] Connection header received
0929 03:18:18.408158 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:18:18.408158 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:18:18.408169 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:18:18.408223 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:18:18.408503 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:18.409169 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:18.409819 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:18.410004 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:18.410057 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:18.412340 (+ 2283us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.178.66:41007: BlockingRecv error: recv got EOF from 127.20.178.66:41007 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Sep 29 03:18:18 dist-test-slave-c9qh krb5kdc[29344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.178.67@KRBTEST.COM for oryx/127.20.178.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:18:18 dist-test-slave-c9qh krb5kdc[29344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.178.67@KRBTEST.COM for oryx/127.20.178.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:18:18.530943 30376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:18.526154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:44981 (local address 127.20.178.67:37343)
0929 03:18:18.526294 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:18.526304 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 03:18:18.526356 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:18:18.526478 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:18:18.526479 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:18:18.526623 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:18:18.526670 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:18.526672 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:18.527289 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:18.527290 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:18.527831 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:18.527832 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:18.528141 (+ 309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:18.528141 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:18.528186 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:18.528788 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:18:18.528793 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:18:18.530811 (+ 2018us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.178.126:44981: Server oryx/127.20.178.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":83,"threads_started":1}
W20260929 03:18:18.530957 30374 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:18:18.526231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.67:37343 (local address 127.20.178.126:44981)
0929 03:18:18.526300 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0929 03:18:18.526303 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:18:18.526346 (+ 43us) server_negotiation.cc:407] Connection header received
0929 03:18:18.526398 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:18:18.526399 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:18:18.526410 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:18:18.526459 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:18:18.526735 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:18.527207 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:18.527905 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:18.528080 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:18.528128 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:18.530910 (+ 2782us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.178.67:37343: BlockingRecv error: recv got EOF from 127.20.178.67:37343 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 29 03:18:19 dist-test-slave-c9qh krb5kdc[29344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.178.65@KRBTEST.COM for oryx/127.20.178.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:18:19 dist-test-slave-c9qh krb5kdc[29344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.178.65@KRBTEST.COM for oryx/127.20.178.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:18:19.291487 30377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:19.286764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:44981 (local address 127.20.178.65:40609)
0929 03:18:19.286919 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:19.286932 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 03:18:19.286995 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:18:19.287127 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:18:19.287128 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:18:19.287284 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:18:19.287333 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:19.287334 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:19.288208 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:19.288209 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:19.288758 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:19.288760 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:19.289075 (+ 315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:19.289076 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:19.289113 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:19.289649 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:18:19.289653 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:18:19.291428 (+ 1775us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.178.126:44981: Server oryx/127.20.178.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":66,"threads_started":1}
W20260929 03:18:19.291724 30378 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:18:19.286825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.65:40609 (local address 127.20.178.126:44981)
0929 03:18:19.286977 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0929 03:18:19.286979 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:18:19.286988 (+ 9us) server_negotiation.cc:407] Connection header received
0929 03:18:19.287049 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:18:19.287049 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:18:19.287060 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:18:19.287109 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:18:19.287394 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:19.288135 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:19.288824 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:19.289012 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:19.289063 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:19.291604 (+ 2541us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.178.65:40609: BlockingRecv error: recv got EOF from 127.20.178.65:40609 (error 108)
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":89,"threads_started":1}
Sep 29 03:18:19 dist-test-slave-c9qh krb5kdc[29344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.178.66@KRBTEST.COM for oryx/127.20.178.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:18:19 dist-test-slave-c9qh krb5kdc[29344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.178.66@KRBTEST.COM for oryx/127.20.178.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:18:19.423183 30379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:19.412847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:44981 (local address 127.20.178.66:36551)
0929 03:18:19.413016 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:19.413028 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 03:18:19.413099 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:18:19.413255 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:18:19.413256 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:18:19.413438 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:18:19.413485 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:19.413486 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:19.414186 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:19.414187 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:19.414718 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:19.414719 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:19.415092 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:19.415093 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:19.415159 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:19.421364 (+ 6205us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:18:19.421369 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:18:19.423129 (+ 1760us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.178.126:44981: Server oryx/127.20.178.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":90,"threads_started":1}
W20260929 03:18:19.423329 30378 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:18:19.412940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.66:36551 (local address 127.20.178.126:44981)
0929 03:18:19.413017 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0929 03:18:19.413020 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:18:19.413081 (+ 61us) server_negotiation.cc:407] Connection header received
0929 03:18:19.413156 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:18:19.413156 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:18:19.413170 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:18:19.413247 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:18:19.413582 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:19.414095 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:19.414819 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:19.415009 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:19.415093 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:19.423236 (+ 8143us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.178.66:36551: BlockingRecv error: recv got EOF from 127.20.178.66:36551 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 29 03:18:19 dist-test-slave-c9qh krb5kdc[29344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.178.67@KRBTEST.COM for oryx/127.20.178.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:18:19 dist-test-slave-c9qh krb5kdc[29344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.178.67@KRBTEST.COM for oryx/127.20.178.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:18:19.535997 30380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:19.531400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:44981 (local address 127.20.178.67:51585)
0929 03:18:19.531527 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:19.531538 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 03:18:19.531600 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:18:19.531737 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:18:19.531737 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:18:19.531887 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:18:19.531940 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:19.531941 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:19.532622 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:19.532623 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:19.533238 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:19.533241 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:19.533553 (+ 312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:19.533554 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:19.533592 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:19.534075 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:18:19.534080 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:18:19.535903 (+ 1823us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.178.126:44981: Server oryx/127.20.178.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":84,"threads_started":1}
W20260929 03:18:19.536048 30378 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:18:19.531449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.67:51585 (local address 127.20.178.126:44981)
0929 03:18:19.531512 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0929 03:18:19.531514 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:18:19.531599 (+ 85us) server_negotiation.cc:407] Connection header received
0929 03:18:19.531647 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:18:19.531648 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:18:19.531660 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:18:19.531709 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:18:19.531999 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:19.532496 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:19.533309 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:19.533489 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:19.533543 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:19.535984 (+ 2441us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.178.67:51585: BlockingRecv error: recv got EOF from 127.20.178.67:51585 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Sep 29 03:18:20 dist-test-slave-c9qh krb5kdc[29344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.178.65@KRBTEST.COM for oryx/127.20.178.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:18:20 dist-test-slave-c9qh krb5kdc[29344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.178.65@KRBTEST.COM for oryx/127.20.178.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:18:20.296614 30381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:20.291942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:44981 (local address 127.20.178.65:52337)
0929 03:18:20.292082 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:20.292102 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 03:18:20.292163 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:18:20.292311 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:18:20.292311 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:18:20.292499 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:18:20.292549 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:20.292551 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:20.293259 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:20.293260 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:20.293910 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:20.293912 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:20.294230 (+ 318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:20.294231 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:20.294272 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:20.294717 (+ 445us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:18:20.294721 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:18:20.296553 (+ 1832us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.178.126:44981: Server oryx/127.20.178.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":88,"threads_started":1}
W20260929 03:18:20.296743 30382 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:18:20.291982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.65:52337 (local address 127.20.178.126:44981)
0929 03:18:20.292119 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0929 03:18:20.292121 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:18:20.292158 (+ 37us) server_negotiation.cc:407] Connection header received
0929 03:18:20.292220 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:18:20.292221 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:18:20.292233 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:18:20.292292 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:18:20.292676 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:20.293185 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:20.293982 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:20.294166 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:20.294216 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:20.296685 (+ 2469us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.178.65:52337: BlockingRecv error: recv got EOF from 127.20.178.65:52337 (error 108)
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":82,"threads_started":1}
Sep 29 03:18:20 dist-test-slave-c9qh krb5kdc[29344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.178.66@KRBTEST.COM for oryx/127.20.178.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:18:20 dist-test-slave-c9qh krb5kdc[29344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.178.66@KRBTEST.COM for oryx/127.20.178.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:18:20.428202 30383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:20.423704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:44981 (local address 127.20.178.66:57399)
0929 03:18:20.423850 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:20.423861 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 03:18:20.423928 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:18:20.424061 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:18:20.424061 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:18:20.424252 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:18:20.424302 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:20.424303 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:20.424954 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:20.424955 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:20.425536 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:20.425538 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:20.425882 (+ 344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:20.425882 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:20.425926 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:20.426383 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:18:20.426388 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:18:20.428123 (+ 1735us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.178.126:44981: Server oryx/127.20.178.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":94,"threads_started":1}
W20260929 03:18:20.428306 30382 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:18:20.423743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.66:57399 (local address 127.20.178.126:44981)
0929 03:18:20.423810 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0929 03:18:20.423812 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:18:20.423931 (+ 119us) server_negotiation.cc:407] Connection header received
0929 03:18:20.423987 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:18:20.423988 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:18:20.423998 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:18:20.424040 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:18:20.424366 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:20.424880 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:20.425618 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:20.425816 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:20.425866 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:20.428234 (+ 2368us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.178.66:57399: BlockingRecv error: recv got EOF from 127.20.178.66:57399 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Sep 29 03:18:20 dist-test-slave-c9qh krb5kdc[29344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.178.67@KRBTEST.COM for oryx/127.20.178.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:18:20 dist-test-slave-c9qh krb5kdc[29344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.178.67@KRBTEST.COM for oryx/127.20.178.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:18:20.541169 30384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:20.536494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:44981 (local address 127.20.178.67:53165)
0929 03:18:20.536624 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:20.536720 (+ 96us) client_negotiation.cc:174] Beginning negotiation
0929 03:18:20.536775 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:18:20.536946 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:18:20.536947 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:18:20.537114 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:18:20.537161 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:20.537163 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:20.537809 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:20.537810 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:20.538343 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:20.538344 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:20.538658 (+ 314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:20.538659 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:20.538700 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:20.539124 (+ 424us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:18:20.539129 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:18:20.541063 (+ 1934us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.178.126:44981: Server oryx/127.20.178.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":80,"threads_started":1}
W20260929 03:18:20.541200 30382 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:18:20.536623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.67:53165 (local address 127.20.178.126:44981)
0929 03:18:20.536708 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0929 03:18:20.536710 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:18:20.536764 (+ 54us) server_negotiation.cc:407] Connection header received
0929 03:18:20.536831 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:18:20.536832 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:18:20.536843 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:18:20.536898 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:18:20.537233 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:20.537712 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:20.538418 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:20.538590 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:20.538636 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:20.541136 (+ 2500us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.178.67:53165: BlockingRecv error: recv got EOF from 127.20.178.67:53165 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 29 03:18:21 dist-test-slave-c9qh krb5kdc[29344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.178.65@KRBTEST.COM for oryx/127.20.178.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:18:21 dist-test-slave-c9qh krb5kdc[29344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.178.65@KRBTEST.COM for oryx/127.20.178.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:18:21.302315 30385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:21.297226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:44981 (local address 127.20.178.65:60261)
0929 03:18:21.297364 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:21.297375 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 03:18:21.297432 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:18:21.297564 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:18:21.297565 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:18:21.297730 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:18:21.297776 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:21.297778 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:21.298403 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:21.298404 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:21.298940 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:21.298942 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:21.299477 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:21.299477 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:21.299516 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:21.300027 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:18:21.300032 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:18:21.302195 (+ 2163us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.178.126:44981: Server oryx/127.20.178.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":86,"threads_started":1}
W20260929 03:18:21.302367 30386 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:18:21.297294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.65:60261 (local address 127.20.178.126:44981)
0929 03:18:21.297439 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0929 03:18:21.297441 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:18:21.297451 (+ 10us) server_negotiation.cc:407] Connection header received
0929 03:18:21.297474 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:18:21.297474 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:18:21.297484 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:18:21.297550 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:18:21.297844 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:21.298333 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:21.299016 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:21.299390 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:21.299470 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:21.302282 (+ 2812us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.178.65:60261: BlockingRecv error: recv got EOF from 127.20.178.65:60261 (error 108)
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":85,"threads_started":1}
Sep 29 03:18:21 dist-test-slave-c9qh krb5kdc[29344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.178.66@KRBTEST.COM for oryx/127.20.178.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:18:21 dist-test-slave-c9qh krb5kdc[29344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.178.66@KRBTEST.COM for oryx/127.20.178.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:18:21.433488 30387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:21.428813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:44981 (local address 127.20.178.66:41507)
0929 03:18:21.428926 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:21.428938 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 03:18:21.429010 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:18:21.429121 (+ 111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:18:21.429121 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:18:21.429302 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:18:21.429345 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:21.429346 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:21.430062 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:21.430063 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:21.430599 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:21.430601 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:21.430943 (+ 342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:21.430944 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:21.430987 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:21.431435 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:18:21.431439 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:18:21.433421 (+ 1982us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.178.126:44981: Server oryx/127.20.178.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":57,"threads_started":1}
W20260929 03:18:21.433598 30386 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:18:21.428883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.66:41507 (local address 127.20.178.126:44981)
0929 03:18:21.428960 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0929 03:18:21.428961 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 03:18:21.428994 (+ 33us) server_negotiation.cc:407] Connection header received
0929 03:18:21.429058 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:18:21.429058 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:18:21.429070 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:18:21.429117 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:18:21.429427 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:21.429976 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:21.430672 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:21.430870 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:21.430929 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:21.433506 (+ 2577us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.178.66:41507: BlockingRecv error: recv got EOF from 127.20.178.66:41507 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 29 03:18:21 dist-test-slave-c9qh krb5kdc[29344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.178.67@KRBTEST.COM for oryx/127.20.178.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:18:21 dist-test-slave-c9qh krb5kdc[29344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.178.67@KRBTEST.COM for oryx/127.20.178.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:18:21.546216 30388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:21.541657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:44981 (local address 127.20.178.67:43875)
0929 03:18:21.541807 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:21.541817 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 03:18:21.541873 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:18:21.542016 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:18:21.542016 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:18:21.542178 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:18:21.542225 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:21.542226 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:21.542882 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:21.542882 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:21.543449 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:21.543450 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:21.543816 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:21.543817 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:21.543859 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:21.544302 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:18:21.544306 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:18:21.546150 (+ 1844us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.178.126:44981: Server oryx/127.20.178.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":87,"threads_started":1}
W20260929 03:18:21.546413 30386 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:18:21.541763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.67:43875 (local address 127.20.178.126:44981)
0929 03:18:21.541811 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0929 03:18:21.541813 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:18:21.541869 (+ 56us) server_negotiation.cc:407] Connection header received
0929 03:18:21.541938 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:18:21.541938 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:18:21.541951 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:18:21.541994 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:18:21.542298 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:21.542812 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:21.543531 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:21.543748 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:21.543803 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:21.546342 (+ 2539us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.178.67:43875: BlockingRecv error: recv got EOF from 127.20.178.67:43875 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
Sep 29 03:18:22 dist-test-slave-c9qh krb5kdc[29344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.178.65@KRBTEST.COM for oryx/127.20.178.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:18:22 dist-test-slave-c9qh krb5kdc[29344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.178.65@KRBTEST.COM for oryx/127.20.178.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:18:22.307909 30389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:22.302928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:44981 (local address 127.20.178.65:46869)
0929 03:18:22.303102 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:22.303112 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 03:18:22.303165 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:18:22.303310 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:18:22.303311 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:18:22.303474 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:18:22.303519 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:22.303521 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:22.304246 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:22.304246 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:22.304843 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:22.304845 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:22.305220 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:22.305221 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:22.305266 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:22.305846 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:18:22.305850 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:18:22.307811 (+ 1961us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.178.126:44981: Server oryx/127.20.178.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":110,"threads_started":1}
W20260929 03:18:22.307974 30390 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:18:22.303032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.65:46869 (local address 127.20.178.126:44981)
0929 03:18:22.303169 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0929 03:18:22.303171 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:18:22.303204 (+ 33us) server_negotiation.cc:407] Connection header received
0929 03:18:22.303227 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:18:22.303228 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:18:22.303239 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:18:22.303294 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:18:22.303598 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:22.304164 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:22.304926 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:22.305147 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:22.305208 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:22.307897 (+ 2689us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.178.65:46869: BlockingRecv error: recv got EOF from 127.20.178.65:46869 (error 108)
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":61,"threads_started":1}
Sep 29 03:18:22 dist-test-slave-c9qh krb5kdc[29344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.178.66@KRBTEST.COM for oryx/127.20.178.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:18:22 dist-test-slave-c9qh krb5kdc[29344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.178.66@KRBTEST.COM for oryx/127.20.178.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:18:22.438766 30391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:22.433968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:44981 (local address 127.20.178.66:56207)
0929 03:18:22.434145 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:22.434161 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 03:18:22.434265 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:18:22.434402 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:18:22.434402 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:18:22.434589 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:18:22.434633 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:22.434635 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:22.435332 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:22.435333 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:22.435888 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:22.435890 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:22.436270 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:22.436271 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:22.436336 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:22.436873 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:18:22.436878 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:18:22.438675 (+ 1797us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.178.126:44981: Server oryx/127.20.178.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":101,"threads_started":1}
W20260929 03:18:22.438918 30390 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:18:22.434084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.66:56207 (local address 127.20.178.126:44981)
0929 03:18:22.434163 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0929 03:18:22.434165 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:18:22.434246 (+ 81us) server_negotiation.cc:407] Connection header received
0929 03:18:22.434325 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:18:22.434326 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:18:22.434339 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:18:22.434387 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:18:22.434720 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:22.435224 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:22.435976 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:22.436180 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:22.436269 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:22.438860 (+ 2591us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.178.66:56207: BlockingRecv error: recv got EOF from 127.20.178.66:56207 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 29 03:18:22 dist-test-slave-c9qh krb5kdc[29344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.178.67@KRBTEST.COM for oryx/127.20.178.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:18:22 dist-test-slave-c9qh krb5kdc[29344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.178.67@KRBTEST.COM for oryx/127.20.178.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:18:22.551756 30392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:22.546720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:44981 (local address 127.20.178.67:36381)
0929 03:18:22.546871 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:22.546891 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 03:18:22.546940 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:18:22.547169 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:18:22.547170 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:18:22.547354 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:18:22.547399 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:22.547401 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:22.548188 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:22.548189 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:22.548996 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:22.549000 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:22.549355 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:22.549356 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:22.549394 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:22.549897 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:18:22.549902 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:18:22.551699 (+ 1797us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.178.126:44981: Server oryx/127.20.178.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":100,"threads_started":1}
W20260929 03:18:22.551829 30390 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:18:22.546948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.67:36381 (local address 127.20.178.126:44981)
0929 03:18:22.547032 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0929 03:18:22.547034 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:18:22.547042 (+ 8us) server_negotiation.cc:407] Connection header received
0929 03:18:22.547065 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:18:22.547066 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:18:22.547077 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:18:22.547158 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:18:22.547487 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:22.548046 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:22.549076 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:22.549282 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:22.549340 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:22.551780 (+ 2440us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.178.67:36381: BlockingRecv error: recv got EOF from 127.20.178.67:36381 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 29 03:18:23 dist-test-slave-c9qh krb5kdc[29344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.178.65@KRBTEST.COM for oryx/127.20.178.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:18:23 dist-test-slave-c9qh krb5kdc[29344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.178.65@KRBTEST.COM for oryx/127.20.178.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:18:23.313199 30393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:23.308528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:44981 (local address 127.20.178.65:58049)
0929 03:18:23.308707 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:23.308717 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 03:18:23.308766 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:18:23.308935 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:18:23.308936 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:18:23.309122 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:18:23.309168 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:23.309169 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:23.309853 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:23.309853 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:23.310394 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:23.310395 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:23.310726 (+ 331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:23.310727 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:23.310771 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:23.311320 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:18:23.311324 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:18:23.313124 (+ 1800us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.178.126:44981: Server oryx/127.20.178.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":93,"threads_started":1}
W20260929 03:18:23.313323 30394 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:18:23.308611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.65:58049 (local address 127.20.178.126:44981)
0929 03:18:23.308750 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0929 03:18:23.308752 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:18:23.308759 (+ 7us) server_negotiation.cc:407] Connection header received
0929 03:18:23.308824 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:18:23.308825 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:18:23.308835 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:18:23.308910 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:18:23.309281 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:23.309781 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:23.310465 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:23.310653 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:23.310703 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:23.313222 (+ 2519us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.178.65:58049: BlockingRecv error: recv got EOF from 127.20.178.65:58049 (error 108)
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":83,"threads_started":1}
Sep 29 03:18:23 dist-test-slave-c9qh krb5kdc[29344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.178.66@KRBTEST.COM for oryx/127.20.178.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:18:23 dist-test-slave-c9qh krb5kdc[29344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.178.66@KRBTEST.COM for oryx/127.20.178.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:18:23.444168 30395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:23.439279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:44981 (local address 127.20.178.66:54739)
0929 03:18:23.439417 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:23.439427 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 03:18:23.439482 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:18:23.439645 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:18:23.439645 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:18:23.439811 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:18:23.439861 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:23.439863 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:23.440545 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:23.440546 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:23.441273 (+ 727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:23.441276 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:23.441703 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:23.441704 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:23.441747 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:23.442217 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:18:23.442221 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:18:23.444050 (+ 1829us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.178.126:44981: Server oryx/127.20.178.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":79,"threads_started":1}
W20260929 03:18:23.444187 30394 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:18:23.439369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.66:54739 (local address 127.20.178.126:44981)
0929 03:18:23.439414 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0929 03:18:23.439416 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:18:23.439467 (+ 51us) server_negotiation.cc:407] Connection header received
0929 03:18:23.439529 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:18:23.439529 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:18:23.439540 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:18:23.439591 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:18:23.439957 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:23.440466 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:23.441422 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:23.441626 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:23.441690 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:23.444127 (+ 2437us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.178.66:54739: BlockingRecv error: recv got EOF from 127.20.178.66:54739 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
Sep 29 03:18:23 dist-test-slave-c9qh krb5kdc[29344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.178.67@KRBTEST.COM for oryx/127.20.178.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:18:23 dist-test-slave-c9qh krb5kdc[29344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.178.67@KRBTEST.COM for oryx/127.20.178.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:18:23.557231 30396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:23.552343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:44981 (local address 127.20.178.67:47737)
0929 03:18:23.552501 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:23.552513 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 03:18:23.552589 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:18:23.552789 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:18:23.552790 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:18:23.552949 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:18:23.552994 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:23.552996 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:23.553638 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:23.553639 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:23.554192 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:23.554194 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:23.554532 (+ 338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:23.554533 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:23.554574 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:23.555156 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:18:23.555161 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:18:23.557121 (+ 1960us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.178.126:44981: Server oryx/127.20.178.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":102,"threads_started":1}
W20260929 03:18:23.557269 30394 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:18:23.552444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.67:47737 (local address 127.20.178.126:44981)
0929 03:18:23.552512 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0929 03:18:23.552514 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:18:23.552591 (+ 77us) server_negotiation.cc:407] Connection header received
0929 03:18:23.552707 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:18:23.552708 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:18:23.552721 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:18:23.552768 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:18:23.553057 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:23.553574 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:23.554270 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:23.554458 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:23.554520 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:23.557204 (+ 2684us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.178.67:47737: BlockingRecv error: recv got EOF from 127.20.178.67:47737 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Sep 29 03:18:24 dist-test-slave-c9qh krb5kdc[29344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.178.65@KRBTEST.COM for oryx/127.20.178.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:18:24 dist-test-slave-c9qh krb5kdc[29344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.178.65@KRBTEST.COM for oryx/127.20.178.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:18:24.318218 30397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:24.313665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:44981 (local address 127.20.178.65:40927)
0929 03:18:24.313824 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:24.313834 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 03:18:24.313885 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:18:24.314019 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:18:24.314020 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:18:24.314184 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:18:24.314232 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:24.314234 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:24.314903 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:24.314904 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:24.315416 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:24.315417 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:24.315813 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:24.315813 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:24.315854 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:24.316360 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:18:24.316365 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:18:24.318163 (+ 1798us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.178.126:44981: Server oryx/127.20.178.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":94,"threads_started":1}
W20260929 03:18:24.318336 30398 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:18:24.313753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.65:40927 (local address 127.20.178.126:44981)
0929 03:18:24.313893 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0929 03:18:24.313894 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 03:18:24.313905 (+ 11us) server_negotiation.cc:407] Connection header received
0929 03:18:24.313934 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:18:24.313934 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:18:24.313945 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:18:24.314000 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:18:24.314331 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:24.314834 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:24.315490 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:24.315715 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:24.315768 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:24.318255 (+ 2487us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.178.65:40927: BlockingRecv error: recv got EOF from 127.20.178.65:40927 (error 108)
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":86,"threads_started":1}
Sep 29 03:18:24 dist-test-slave-c9qh krb5kdc[29344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.178.66@KRBTEST.COM for oryx/127.20.178.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:18:24 dist-test-slave-c9qh krb5kdc[29344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.178.66@KRBTEST.COM for oryx/127.20.178.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:18:24.454895 30399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:24.444747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:44981 (local address 127.20.178.66:59843)
0929 03:18:24.444884 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:24.444894 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 03:18:24.444956 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:18:24.445091 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:18:24.445092 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:18:24.445257 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:18:24.445303 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:24.445304 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:24.445934 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:24.445935 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:24.446445 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:24.446446 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:24.446785 (+ 339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:24.446786 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:24.446824 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:24.452793 (+ 5969us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:18:24.452799 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:18:24.454799 (+ 2000us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.178.126:44981: Server oryx/127.20.178.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":88,"threads_started":1}
W20260929 03:18:24.454957 30398 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:18:24.444795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.66:59843 (local address 127.20.178.126:44981)
0929 03:18:24.444865 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0929 03:18:24.444867 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:18:24.444956 (+ 89us) server_negotiation.cc:407] Connection header received
0929 03:18:24.445005 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:18:24.445006 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:18:24.445017 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:18:24.445066 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:18:24.445370 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:24.445867 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:24.446518 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:24.446721 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:24.446775 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:24.454895 (+ 8120us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.178.66:59843: BlockingRecv error: recv got EOF from 127.20.178.66:59843 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Sep 29 03:18:24 dist-test-slave-c9qh krb5kdc[29344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.178.67@KRBTEST.COM for oryx/127.20.178.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:18:24 dist-test-slave-c9qh krb5kdc[29344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.178.67@KRBTEST.COM for oryx/127.20.178.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:18:24.562599 30400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:24.557710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:44981 (local address 127.20.178.67:48027)
0929 03:18:24.557917 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:24.557927 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 03:18:24.558008 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:18:24.558291 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:18:24.558293 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:18:24.558505 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:18:24.558552 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:24.558555 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:24.559269 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:24.559270 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:24.559785 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:24.559786 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:24.560131 (+ 345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:24.560132 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:24.560171 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:24.560616 (+ 445us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:18:24.560620 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:18:24.562500 (+ 1880us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.178.126:44981: Server oryx/127.20.178.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":115,"threads_started":1}
W20260929 03:18:24.562656 30398 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:18:24.557829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.67:48027 (local address 127.20.178.126:44981)
0929 03:18:24.557896 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0929 03:18:24.557899 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:18:24.558023 (+ 124us) server_negotiation.cc:407] Connection header received
0929 03:18:24.558072 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:18:24.558072 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:18:24.558092 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:18:24.558157 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:18:24.558662 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:24.559183 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:24.559871 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:24.560061 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:24.560121 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:24.562586 (+ 2465us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.178.67:48027: BlockingRecv error: recv got EOF from 127.20.178.67:48027 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Sep 29 03:18:25 dist-test-slave-c9qh krb5kdc[29344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.178.65@KRBTEST.COM for oryx/127.20.178.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:18:25 dist-test-slave-c9qh krb5kdc[29344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.178.65@KRBTEST.COM for oryx/127.20.178.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:18:25.323101 30401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:25.318728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:44981 (local address 127.20.178.65:49213)
0929 03:18:25.318864 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:25.318874 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 03:18:25.318924 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:18:25.319065 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:18:25.319066 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:18:25.319225 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:18:25.319273 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:25.319274 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:25.319904 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:25.319905 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:25.320437 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:25.320439 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:25.320778 (+ 339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:25.320779 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:25.320823 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:25.321246 (+ 423us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:18:25.321250 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:18:25.323030 (+ 1780us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.178.126:44981: Server oryx/127.20.178.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":90,"threads_started":1}
W20260929 03:18:25.323252 30402 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:18:25.318820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.65:49213 (local address 127.20.178.126:44981)
0929 03:18:25.318963 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0929 03:18:25.318965 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:18:25.318973 (+ 8us) server_negotiation.cc:407] Connection header received
0929 03:18:25.318993 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:18:25.318993 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:18:25.319003 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:18:25.319061 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:18:25.319336 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:25.319834 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:25.320506 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:25.320709 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:25.320759 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:25.323130 (+ 2371us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.178.65:49213: BlockingRecv error: recv got EOF from 127.20.178.65:49213 (error 108)
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":88,"threads_started":1}
Sep 29 03:18:25 dist-test-slave-c9qh krb5kdc[29344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.178.66@KRBTEST.COM for oryx/127.20.178.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:18:25 dist-test-slave-c9qh krb5kdc[29344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.178.66@KRBTEST.COM for oryx/127.20.178.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:18:25.460158 30403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:25.455410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:44981 (local address 127.20.178.66:48695)
0929 03:18:25.455543 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:25.455553 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 03:18:25.455616 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:18:25.455803 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:18:25.455804 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:18:25.455974 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:18:25.456022 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:25.456025 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:25.456808 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:25.456809 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:25.457353 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:25.457355 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:25.457731 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:25.457732 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:25.457800 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:25.458313 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:18:25.458318 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:18:25.460051 (+ 1733us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.178.126:44981: Server oryx/127.20.178.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":67,"threads_started":1}
W20260929 03:18:25.460198 30402 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:18:25.455476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.66:48695 (local address 127.20.178.126:44981)
0929 03:18:25.455570 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0929 03:18:25.455572 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:18:25.455602 (+ 30us) server_negotiation.cc:407] Connection header received
0929 03:18:25.455698 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:18:25.455699 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:18:25.455711 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:18:25.455794 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:18:25.456105 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:25.456613 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:25.457452 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:25.457651 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:25.457723 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:25.460134 (+ 2411us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.178.66:48695: BlockingRecv error: recv got EOF from 127.20.178.66:48695 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 03:18:25.487865 21193 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuoaHbX/build/release/bin/kudu with pid 29927
I20260929 03:18:25.492578 21193 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuoaHbX/build/release/bin/kudu with pid 30079
I20260929 03:18:25.496610 21193 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuoaHbX/build/release/bin/kudu with pid 30212
I20260929 03:18:25.501133 21193 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuoaHbX/build/release/bin/kudu with pid 29857
2026-09-29T03:18:25Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17398 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-29T03:18:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:18:25Z Disabled control of system clock
I20260929 03:18:25.528162 21193 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuoaHbX/build/release/bin/kudu
/tmp/dist-test-taskuoaHbX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790651852476208-21193-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790651852476208-21193-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790651852476208-21193-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790651852476208-21193-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.178.126:38853
--webserver_interface=127.20.178.126
--webserver_port=0
--builtin_ntp_servers=127.20.178.84:38667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.178.126:38853
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260929 03:18:25.607990 30407 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:18:25.608192 30407 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:18:25.608212 30407 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:18:25.609352 30407 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:18:25.609388 30407 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:18:25.609401 30407 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:18:25.609413 30407 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:18:25.609426 30407 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:18:25.610677 30407 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.178.84:38667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790651852476208-21193-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790651852476208-21193-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.178.126:38853
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.178.126:38853
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790651852476208-21193-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.178.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30407
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790651852476208-21193-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:18:25.610860 30407 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:18:25.611083 30407 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 03:18:25.616166 30407 server_base.cc:1034] running on GCE node
W20260929 03:18:25.616150 30413 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:18:25.616160 30412 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:18:25.616150 30415 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:18:25.616698 30407 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:18:25.616953 30407 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:18:25.618117 30407 hybrid_clock.cc:648] HybridClock initialized: now 1790651905618093 us; error 38 us; skew 500 ppm
I20260929 03:18:25.619315 30407 webserver.cc:460] Webserver started at http://127.20.178.126:40129/ using document root <none> and password file <none>
I20260929 03:18:25.619544 30407 fs_manager.cc:359] Metadata directory not provided
I20260929 03:18:25.619611 30407 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:18:25.619779 30407 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:18:25.620886 30407 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790651852476208-21193-0/minicluster-data/master-0/data/instance:
uuid: "ac1c8507555a4791b3097b6c44cc3f17"
format_stamp: "Formatted at 2026-09-29 03:18:25 on dist-test-slave-c9qh"
server_key: "e5b2f64e6ced6404b7a66b8f7e780631"
server_key_iv: "1b4b5d54a10746a948483e62c622dfd2"
server_key_version: "encryptionkey@0"
I20260929 03:18:25.621234 30407 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790651852476208-21193-0/minicluster-data/master-0/wal/instance:
uuid: "ac1c8507555a4791b3097b6c44cc3f17"
format_stamp: "Formatted at 2026-09-29 03:18:25 on dist-test-slave-c9qh"
server_key: "e5b2f64e6ced6404b7a66b8f7e780631"
server_key_iv: "1b4b5d54a10746a948483e62c622dfd2"
server_key_version: "encryptionkey@0"
I20260929 03:18:25.622552 30407 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 03:18:25.623278 30421 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:18:25.623456 30407 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 03:18:25.623558 30407 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790651852476208-21193-0/minicluster-data/master-0/data,/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790651852476208-21193-0/minicluster-data/master-0/wal
uuid: "ac1c8507555a4791b3097b6c44cc3f17"
format_stamp: "Formatted at 2026-09-29 03:18:25 on dist-test-slave-c9qh"
server_key: "e5b2f64e6ced6404b7a66b8f7e780631"
server_key_iv: "1b4b5d54a10746a948483e62c622dfd2"
server_key_version: "encryptionkey@0"
I20260929 03:18:25.623629 30407 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790651852476208-21193-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790651852476208-21193-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790651852476208-21193-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:18:25.636158 30407 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:18:25.636454 30407 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:18:25.636601 30407 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:18:25.640614 30407 rpc_server.cc:307] RPC server started. Bound to: 127.20.178.126:38853
I20260929 03:18:25.640697 30473 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.178.126:38853 every 8 connection(s)
I20260929 03:18:25.641081 30407 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790651852476208-21193-0/minicluster-data/master-0/data/info.pb
I20260929 03:18:25.641748 30474 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:18:25.642038 21193 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuoaHbX/build/release/bin/kudu as pid 30407
I20260929 03:18:25.642112 21193 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790651852476208-21193-0/minicluster-data/master-0/wal/instance
I20260929 03:18:25.642308 21193 external_mini_cluster.cc:1428] Setting key cf98dc6446c74e2e9d8c41a554522c1b
I20260929 03:18:25.645408 30474 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac1c8507555a4791b3097b6c44cc3f17: Bootstrap starting.
I20260929 03:18:25.646106 30474 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ac1c8507555a4791b3097b6c44cc3f17: Neither blocks nor log segments found. Creating new log.
I20260929 03:18:25.646397 30474 log.cc:824] T 00000000000000000000000000000000 P ac1c8507555a4791b3097b6c44cc3f17: Log is configured to *not* fsync() on all Append() calls
I20260929 03:18:25.647060 30474 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac1c8507555a4791b3097b6c44cc3f17: No bootstrap required, opened a new log
I20260929 03:18:25.648183 30477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:25.642833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36266 (local address 127.20.178.126:38853)
0929 03:18:25.643052 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0929 03:18:25.643061 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 03:18:25.643092 (+ 31us) server_negotiation.cc:407] Connection header received
0929 03:18:25.643627 (+ 535us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:18:25.643636 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:18:25.643718 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:18:25.643866 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:18:25.644027 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:25.645383 (+ 1356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:25.646274 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:25.646713 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:25.646852 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:25.647588 (+ 736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:18:25.647592 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:18:25.647592 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:18:25.647761 (+ 169us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:18:25.647793 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:18:25.647803 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:18:25.647869 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0929 03:18:25.647950 (+ 81us) server_negotiation.cc:299] Negotiation successful
0929 03:18:25.648028 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":91,"threads_started":1}
I20260929 03:18:25.649080 30474 raft_consensus.cc:348] T 00000000000000000000000000000000 P ac1c8507555a4791b3097b6c44cc3f17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac1c8507555a4791b3097b6c44cc3f17" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 38853 } }
I20260929 03:18:25.649224 30474 raft_consensus.cc:374] T 00000000000000000000000000000000 P ac1c8507555a4791b3097b6c44cc3f17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:18:25.649266 30474 raft_consensus.cc:729] T 00000000000000000000000000000000 P ac1c8507555a4791b3097b6c44cc3f17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac1c8507555a4791b3097b6c44cc3f17, State: Initialized, Role: FOLLOWER
I20260929 03:18:25.649380 30474 consensus_queue.cc:260] T 00000000000000000000000000000000 P ac1c8507555a4791b3097b6c44cc3f17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac1c8507555a4791b3097b6c44cc3f17" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 38853 } }
I20260929 03:18:25.649457 30474 raft_consensus.cc:388] T 00000000000000000000000000000000 P ac1c8507555a4791b3097b6c44cc3f17 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:18:25.649497 30474 raft_consensus.cc:482] T 00000000000000000000000000000000 P ac1c8507555a4791b3097b6c44cc3f17 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:18:25.649541 30474 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ac1c8507555a4791b3097b6c44cc3f17 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:18:25.650020 30474 raft_consensus.cc:504] T 00000000000000000000000000000000 P ac1c8507555a4791b3097b6c44cc3f17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac1c8507555a4791b3097b6c44cc3f17" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 38853 } }
I20260929 03:18:25.650173 30474 leader_election.cc:302] T 00000000000000000000000000000000 P ac1c8507555a4791b3097b6c44cc3f17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ac1c8507555a4791b3097b6c44cc3f17; no voters:
I20260929 03:18:25.650367 30474 leader_election.cc:288] T 00000000000000000000000000000000 P ac1c8507555a4791b3097b6c44cc3f17 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:18:25.650419 30479 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ac1c8507555a4791b3097b6c44cc3f17 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:18:25.650585 30479 raft_consensus.cc:686] T 00000000000000000000000000000000 P ac1c8507555a4791b3097b6c44cc3f17 [term 1 LEADER]: Becoming Leader. State: Replica: ac1c8507555a4791b3097b6c44cc3f17, State: Running, Role: LEADER
I20260929 03:18:25.650612 30474 sys_catalog.cc:564] T 00000000000000000000000000000000 P ac1c8507555a4791b3097b6c44cc3f17 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:18:25.650666 30479 consensus_queue.cc:237] T 00000000000000000000000000000000 P ac1c8507555a4791b3097b6c44cc3f17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac1c8507555a4791b3097b6c44cc3f17" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 38853 } }
I20260929 03:18:25.651039 30481 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac1c8507555a4791b3097b6c44cc3f17 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ac1c8507555a4791b3097b6c44cc3f17. Latest consensus state: current_term: 1 leader_uuid: "ac1c8507555a4791b3097b6c44cc3f17" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac1c8507555a4791b3097b6c44cc3f17" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 38853 } } }
I20260929 03:18:25.651129 30481 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac1c8507555a4791b3097b6c44cc3f17 [sys.catalog]: This master's current role is: LEADER
I20260929 03:18:25.651032 30480 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac1c8507555a4791b3097b6c44cc3f17 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ac1c8507555a4791b3097b6c44cc3f17" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac1c8507555a4791b3097b6c44cc3f17" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 38853 } } }
I20260929 03:18:25.651264 30480 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac1c8507555a4791b3097b6c44cc3f17 [sys.catalog]: This master's current role is: LEADER
I20260929 03:18:25.651404 30487 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:18:25.651849 30487 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:18:25.653478 30487 catalog_manager.cc:1347] Generated new cluster ID: 77ed8384f8b4488091331271bb5aaf51
I20260929 03:18:25.653532 30487 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:18:25.664184 30487 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:18:25.664907 30487 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:18:25.668257 30487 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ac1c8507555a4791b3097b6c44cc3f17: Generated new TSK 0
I20260929 03:18:25.668416 30487 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 03:18:25.675429 21193 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuoaHbX/build/release/bin/kudu
/tmp/dist-test-taskuoaHbX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790651852476208-21193-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790651852476208-21193-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790651852476208-21193-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790651852476208-21193-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.178.65:0
--local_ip_for_outbound_sockets=127.20.178.65
--webserver_interface=127.20.178.65
--webserver_port=0
--tserver_master_addrs=127.20.178.126:38853
--builtin_ntp_servers=127.20.178.84:38667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260929 03:18:25.765393 30498 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:18:25.765610 30498 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:18:25.765645 30498 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:18:25.766790 30498 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:18:25.766849 30498 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:18:25.766893 30498 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.178.65
I20260929 03:18:25.768213 30498 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.178.84:38667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790651852476208-21193-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790651852476208-21193-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.178.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790651852476208-21193-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.178.65
--webserver_port=0
--tserver_master_addrs=127.20.178.126:38853
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30498
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.178.65
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790651852476208-21193-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:18:25.768474 30498 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:18:25.768813 30498 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:18:25.773595 30503 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:18:25.773630 30506 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:18:25.773730 30498 server_base.cc:1034] running on GCE node
W20260929 03:18:25.773850 30504 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:18:25.774053 30498 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:18:25.774313 30498 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:18:25.775482 30498 hybrid_clock.cc:648] HybridClock initialized: now 1790651905775467 us; error 30 us; skew 500 ppm
I20260929 03:18:25.776846 30498 webserver.cc:460] Webserver started at http://127.20.178.65:43589/ using document root <none> and password file <none>
I20260929 03:18:25.777073 30498 fs_manager.cc:359] Metadata directory not provided
I20260929 03:18:25.777154 30498 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:18:25.777297 30498 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:18:25.778316 30498 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790651852476208-21193-0/minicluster-data/ts-0/data/instance:
uuid: "1bcb227a166748e983438ef3f46197f7"
format_stamp: "Formatted at 2026-09-29 03:18:25 on dist-test-slave-c9qh"
server_key: "e090eca271607312711e4944d5e64497"
server_key_iv: "ae6c925105feb7c6b2d8c0a0cd35eccd"
server_key_version: "encryptionkey@0"
I20260929 03:18:25.778679 30498 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790651852476208-21193-0/minicluster-data/ts-0/wal/instance:
uuid: "1bcb227a166748e983438ef3f46197f7"
format_stamp: "Formatted at 2026-09-29 03:18:25 on dist-test-slave-c9qh"
server_key: "e090eca271607312711e4944d5e64497"
server_key_iv: "ae6c925105feb7c6b2d8c0a0cd35eccd"
server_key_version: "encryptionkey@0"
I20260929 03:18:25.780030 30498 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 03:18:25.780874 30512 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:18:25.781055 30498 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 03:18:25.781152 30498 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790651852476208-21193-0/minicluster-data/ts-0/data,/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790651852476208-21193-0/minicluster-data/ts-0/wal
uuid: "1bcb227a166748e983438ef3f46197f7"
format_stamp: "Formatted at 2026-09-29 03:18:25 on dist-test-slave-c9qh"
server_key: "e090eca271607312711e4944d5e64497"
server_key_iv: "ae6c925105feb7c6b2d8c0a0cd35eccd"
server_key_version: "encryptionkey@0"
I20260929 03:18:25.781240 30498 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790651852476208-21193-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790651852476208-21193-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790651852476208-21193-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:18:25.826095 30498 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:18:25.826509 30498 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:18:25.826678 30498 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:18:25.826929 30498 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:18:25.827270 30498 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:18:25.827327 30498 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:18:25.827371 30498 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:18:25.827399 30498 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:18:25.833099 30498 rpc_server.cc:307] RPC server started. Bound to: 127.20.178.65:35121
I20260929 03:18:25.833182 30625 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.178.65:35121 every 8 connection(s)
I20260929 03:18:25.833508 30498 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790651852476208-21193-0/minicluster-data/ts-0/data/info.pb
I20260929 03:18:25.833932 21193 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuoaHbX/build/release/bin/kudu as pid 30498
I20260929 03:18:25.834020 21193 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790651852476208-21193-0/minicluster-data/ts-0/wal/instance
I20260929 03:18:25.834218 21193 external_mini_cluster.cc:1428] Setting key cabac6885b4a59385b34636effcc6ebd
I20260929 03:18:25.835270 21193 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuoaHbX/build/release/bin/kudu
/tmp/dist-test-taskuoaHbX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790651852476208-21193-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790651852476208-21193-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790651852476208-21193-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790651852476208-21193-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.178.66:0
--local_ip_for_outbound_sockets=127.20.178.66
--webserver_interface=127.20.178.66
--webserver_port=0
--tserver_master_addrs=127.20.178.126:38853
--builtin_ntp_servers=127.20.178.84:38667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260929 03:18:25.839519 30477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:25.834312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.65:48595 (local address 127.20.178.126:38853)
0929 03:18:25.834381 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0929 03:18:25.834383 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:18:25.834645 (+ 262us) server_negotiation.cc:407] Connection header received
0929 03:18:25.835278 (+ 633us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:18:25.835280 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:18:25.835296 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:18:25.835365 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:18:25.836075 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:25.836782 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:25.837980 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:25.838307 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:25.838384 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:25.839219 (+ 835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:18:25.839221 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:18:25.839222 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:18:25.839227 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:18:25.839246 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:18:25.839250 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:18:25.839313 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0929 03:18:25.839370 (+ 57us) server_negotiation.cc:299] Negotiation successful
0929 03:18:25.839401 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260929 03:18:25.839629 30628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:25.834327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:38853 (local address 127.20.178.65:48595)
0929 03:18:25.834527 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:25.834568 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 03:18:25.835174 (+ 606us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:18:25.835448 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:18:25.835452 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:18:25.835462 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:18:25.835981 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:25.835986 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:25.836878 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:25.836879 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:25.837890 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:25.837893 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:25.838396 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:25.838397 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:25.838503 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:25.839073 (+ 570us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 03:18:25.839078 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:18:25.839107 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 03:18:25.839129 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:18:25.839316 (+ 187us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:18:25.839318 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:18:25.839321 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 03:18:25.839368 (+ 47us) client_negotiation.cc:240] Negotiation successful
0929 03:18:25.839443 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":88,"threads_started":1}
I20260929 03:18:25.840154 30626 heartbeater.cc:347] Connected to a master server at 127.20.178.126:38853
I20260929 03:18:25.840263 30626 heartbeater.cc:464] Registering TS with master...
I20260929 03:18:25.840435 30626 heartbeater.cc:511] Master 127.20.178.126:38853 requested a full tablet report, sending...
I20260929 03:18:25.840945 30438 ts_manager.cc:194] Registered new tserver with Master: 1bcb227a166748e983438ef3f46197f7 (127.20.178.65:35121)
I20260929 03:18:25.841547 30438 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.178.65:48595
W20260929 03:18:25.915994 30629 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:18:25.916204 30629 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:18:25.916225 30629 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:18:25.917404 30629 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:18:25.917444 30629 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:18:25.917469 30629 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.178.66
I20260929 03:18:25.918792 30629 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.178.84:38667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790651852476208-21193-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790651852476208-21193-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.178.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790651852476208-21193-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.178.66
--webserver_port=0
--tserver_master_addrs=127.20.178.126:38853
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30629
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.178.66
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790651852476208-21193-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:18:25.919025 30629 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:18:25.919315 30629 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:18:25.924435 30634 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:18:25.924437 30637 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:18:25.924437 30635 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:18:25.924726 30629 server_base.cc:1034] running on GCE node
I20260929 03:18:25.924957 30629 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:18:25.925201 30629 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:18:25.926373 30629 hybrid_clock.cc:648] HybridClock initialized: now 1790651905926357 us; error 26 us; skew 500 ppm
I20260929 03:18:25.927888 30629 webserver.cc:460] Webserver started at http://127.20.178.66:38851/ using document root <none> and password file <none>
I20260929 03:18:25.928161 30629 fs_manager.cc:359] Metadata directory not provided
I20260929 03:18:25.928236 30629 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:18:25.928381 30629 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:18:25.929543 30629 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790651852476208-21193-0/minicluster-data/ts-1/data/instance:
uuid: "a52ae65683f8420bb7bcaefca2147b9b"
format_stamp: "Formatted at 2026-09-29 03:18:25 on dist-test-slave-c9qh"
server_key: "2a76bf80580485b10923686833fba454"
server_key_iv: "d749ab2f652d57f19374fbbb2c4c45da"
server_key_version: "encryptionkey@0"
I20260929 03:18:25.929957 30629 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790651852476208-21193-0/minicluster-data/ts-1/wal/instance:
uuid: "a52ae65683f8420bb7bcaefca2147b9b"
format_stamp: "Formatted at 2026-09-29 03:18:25 on dist-test-slave-c9qh"
server_key: "2a76bf80580485b10923686833fba454"
server_key_iv: "d749ab2f652d57f19374fbbb2c4c45da"
server_key_version: "encryptionkey@0"
I20260929 03:18:25.931296 30629 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260929 03:18:25.932154 30643 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:18:25.932374 30629 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260929 03:18:25.932484 30629 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790651852476208-21193-0/minicluster-data/ts-1/data,/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790651852476208-21193-0/minicluster-data/ts-1/wal
uuid: "a52ae65683f8420bb7bcaefca2147b9b"
format_stamp: "Formatted at 2026-09-29 03:18:25 on dist-test-slave-c9qh"
server_key: "2a76bf80580485b10923686833fba454"
server_key_iv: "d749ab2f652d57f19374fbbb2c4c45da"
server_key_version: "encryptionkey@0"
I20260929 03:18:25.932570 30629 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790651852476208-21193-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790651852476208-21193-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790651852476208-21193-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:18:25.941684 30629 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:18:25.941946 30629 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:18:25.942086 30629 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:18:25.942296 30629 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:18:25.942616 30629 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:18:25.942670 30629 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:18:25.942711 30629 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:18:25.942739 30629 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:18:25.948622 30629 rpc_server.cc:307] RPC server started. Bound to: 127.20.178.66:40691
I20260929 03:18:25.948710 30756 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.178.66:40691 every 8 connection(s)
I20260929 03:18:25.949040 30629 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790651852476208-21193-0/minicluster-data/ts-1/data/info.pb
I20260929 03:18:25.950148 21193 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuoaHbX/build/release/bin/kudu as pid 30629
I20260929 03:18:25.950243 21193 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790651852476208-21193-0/minicluster-data/ts-1/wal/instance
I20260929 03:18:25.950443 21193 external_mini_cluster.cc:1428] Setting key 005c95aa722eaf9b2309424219d18e7e
I20260929 03:18:25.951557 21193 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuoaHbX/build/release/bin/kudu
/tmp/dist-test-taskuoaHbX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790651852476208-21193-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790651852476208-21193-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790651852476208-21193-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790651852476208-21193-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.178.67:0
--local_ip_for_outbound_sockets=127.20.178.67
--webserver_interface=127.20.178.67
--webserver_port=0
--tserver_master_addrs=127.20.178.126:38853
--builtin_ntp_servers=127.20.178.84:38667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260929 03:18:25.955255 30477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:25.949760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.66:34913 (local address 127.20.178.126:38853)
0929 03:18:25.949803 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0929 03:18:25.949805 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:18:25.950066 (+ 261us) server_negotiation.cc:407] Connection header received
0929 03:18:25.950845 (+ 779us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:18:25.950846 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:18:25.950857 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:18:25.950928 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:18:25.951838 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:25.952713 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:25.953801 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:25.954014 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:25.954110 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:25.954983 (+ 873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:18:25.954985 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:18:25.954985 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:18:25.954990 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:18:25.955005 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:18:25.955009 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:18:25.955062 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0929 03:18:25.955144 (+ 82us) server_negotiation.cc:299] Negotiation successful
0929 03:18:25.955171 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260929 03:18:25.955473 30759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:25.949757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:38853 (local address 127.20.178.66:34913)
0929 03:18:25.949912 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:25.949943 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0929 03:18:25.950658 (+ 715us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:18:25.951004 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:18:25.951008 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:18:25.951019 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:18:25.951733 (+ 714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:25.951740 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:25.952824 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:25.952825 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:25.953709 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:25.953711 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:25.954114 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:25.954115 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:25.954255 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:25.954846 (+ 591us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 03:18:25.954846 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:18:25.954873 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 03:18:25.954895 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:18:25.955074 (+ 179us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:18:25.955079 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:18:25.955082 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 03:18:25.955149 (+ 67us) client_negotiation.cc:240] Negotiation successful
0929 03:18:25.955247 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":66,"threads_started":1}
I20260929 03:18:25.955986 30757 heartbeater.cc:347] Connected to a master server at 127.20.178.126:38853
I20260929 03:18:25.956089 30757 heartbeater.cc:464] Registering TS with master...
I20260929 03:18:25.956297 30757 heartbeater.cc:511] Master 127.20.178.126:38853 requested a full tablet report, sending...
I20260929 03:18:25.956717 30438 ts_manager.cc:194] Registered new tserver with Master: a52ae65683f8420bb7bcaefca2147b9b (127.20.178.66:40691)
I20260929 03:18:25.957254 30438 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.178.66:34913
W20260929 03:18:26.033067 30760 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:18:26.033288 30760 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:18:26.033309 30760 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:18:26.034427 30760 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:18:26.034464 30760 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:18:26.034488 30760 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.178.67
I20260929 03:18:26.035887 30760 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.178.84:38667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790651852476208-21193-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790651852476208-21193-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.178.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790651852476208-21193-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.178.67
--webserver_port=0
--tserver_master_addrs=127.20.178.126:38853
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30760
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.178.67
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790651852476208-21193-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:18:26.036119 30760 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:18:26.036455 30760 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:18:26.041507 30768 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:18:26.041514 30766 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:18:26.041559 30765 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:18:26.041738 30760 server_base.cc:1034] running on GCE node
I20260929 03:18:26.041939 30760 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:18:26.042176 30760 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:18:26.043344 30760 hybrid_clock.cc:648] HybridClock initialized: now 1790651906043327 us; error 30 us; skew 500 ppm
I20260929 03:18:26.044605 30760 webserver.cc:460] Webserver started at http://127.20.178.67:42813/ using document root <none> and password file <none>
I20260929 03:18:26.044883 30760 fs_manager.cc:359] Metadata directory not provided
I20260929 03:18:26.044955 30760 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:18:26.045113 30760 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:18:26.046243 30760 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790651852476208-21193-0/minicluster-data/ts-2/data/instance:
uuid: "a8c39ae5390f4545ab223b3b2400e607"
format_stamp: "Formatted at 2026-09-29 03:18:26 on dist-test-slave-c9qh"
server_key: "3de8b7c6e92dde49b1fce383a7e38dcf"
server_key_iv: "811b35fac1b316ab7e4c654427bb9106"
server_key_version: "encryptionkey@0"
I20260929 03:18:26.046628 30760 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790651852476208-21193-0/minicluster-data/ts-2/wal/instance:
uuid: "a8c39ae5390f4545ab223b3b2400e607"
format_stamp: "Formatted at 2026-09-29 03:18:26 on dist-test-slave-c9qh"
server_key: "3de8b7c6e92dde49b1fce383a7e38dcf"
server_key_iv: "811b35fac1b316ab7e4c654427bb9106"
server_key_version: "encryptionkey@0"
I20260929 03:18:26.047930 30760 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 03:18:26.048748 30774 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:18:26.048928 30760 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 03:18:26.049023 30760 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790651852476208-21193-0/minicluster-data/ts-2/data,/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790651852476208-21193-0/minicluster-data/ts-2/wal
uuid: "a8c39ae5390f4545ab223b3b2400e607"
format_stamp: "Formatted at 2026-09-29 03:18:26 on dist-test-slave-c9qh"
server_key: "3de8b7c6e92dde49b1fce383a7e38dcf"
server_key_iv: "811b35fac1b316ab7e4c654427bb9106"
server_key_version: "encryptionkey@0"
I20260929 03:18:26.049106 30760 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790651852476208-21193-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790651852476208-21193-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790651852476208-21193-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:18:26.074743 30760 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:18:26.075088 30760 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:18:26.075253 30760 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:18:26.075495 30760 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:18:26.075839 30760 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:18:26.075896 30760 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:18:26.075940 30760 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:18:26.075985 30760 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:18:26.081640 30760 rpc_server.cc:307] RPC server started. Bound to: 127.20.178.67:45125
I20260929 03:18:26.081693 30887 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.178.67:45125 every 8 connection(s)
I20260929 03:18:26.082041 30760 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790651852476208-21193-0/minicluster-data/ts-2/data/info.pb
I20260929 03:18:26.086570 21193 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuoaHbX/build/release/bin/kudu as pid 30760
I20260929 03:18:26.086665 21193 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790651852476208-21193-0/minicluster-data/ts-2/wal/instance
I20260929 03:18:26.086848 21193 external_mini_cluster.cc:1428] Setting key 17c29decc307f4639bd6c9a98dc9a7e5
I20260929 03:18:26.087270 30477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:26.082688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.67:42469 (local address 127.20.178.126:38853)
0929 03:18:26.082777 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0929 03:18:26.082780 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:18:26.082990 (+ 210us) server_negotiation.cc:407] Connection header received
0929 03:18:26.083446 (+ 456us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:18:26.083447 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:18:26.083457 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:18:26.083509 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:18:26.084133 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:26.084770 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:26.085847 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:26.086027 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:26.086078 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:26.086996 (+ 918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:18:26.086998 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:18:26.086998 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:18:26.087004 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:18:26.087023 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:18:26.087027 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:18:26.087088 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0929 03:18:26.087162 (+ 74us) server_negotiation.cc:299] Negotiation successful
0929 03:18:26.087188 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 03:18:26.087515 30890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:26.082667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:38853 (local address 127.20.178.67:42469)
0929 03:18:26.082864 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:26.082897 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0929 03:18:26.083368 (+ 471us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:18:26.083561 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:18:26.083564 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:18:26.083571 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:18:26.084057 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:26.084061 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:26.084849 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:26.084850 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:26.085780 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:26.085782 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:26.086100 (+ 318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:26.086101 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:26.086221 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:26.086818 (+ 597us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 03:18:26.086818 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:18:26.086845 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 03:18:26.086867 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:18:26.087089 (+ 222us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:18:26.087094 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:18:26.087097 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 03:18:26.087185 (+ 88us) client_negotiation.cc:240] Negotiation successful
0929 03:18:26.087283 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":99,"threads_started":1}
I20260929 03:18:26.088006 30888 heartbeater.cc:347] Connected to a master server at 127.20.178.126:38853
I20260929 03:18:26.088109 30888 heartbeater.cc:464] Registering TS with master...
I20260929 03:18:26.088296 30888 heartbeater.cc:511] Master 127.20.178.126:38853 requested a full tablet report, sending...
I20260929 03:18:26.088680 30438 ts_manager.cc:194] Registered new tserver with Master: a8c39ae5390f4545ab223b3b2400e607 (127.20.178.67:45125)
I20260929 03:18:26.089054 30438 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.178.67:42469
I20260929 03:18:26.098500 21193 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260929 03:18:26.100263 30477 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:18:26.099796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36278 (local address 127.20.178.126:38853)
0929 03:18:26.099843 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0929 03:18:26.099845 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:18:26.099980 (+ 135us) server_negotiation.cc:407] Connection header received
0929 03:18:26.100014 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:18:26.100015 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:18:26.100024 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:18:26.100062 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:18:26.100206 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36278: BlockingRecv error: recv got EOF from 127.0.0.1:36278 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
I20260929 03:18:26.100344 21193 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuoaHbX/build/release/bin/kudu with pid 30498
I20260929 03:18:26.105072 21193 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuoaHbX/build/release/bin/kudu with pid 30629
I20260929 03:18:26.109668 21193 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuoaHbX/build/release/bin/kudu with pid 30760
I20260929 03:18:26.113713 21193 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuoaHbX/build/release/bin/kudu with pid 30407
2026-09-29T03:18:26Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (611 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-29T03:18:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:18:26Z Disabled control of system clock
I20260929 03:18:26.140273 21193 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuoaHbX/build/release/bin/kudu
/tmp/dist-test-taskuoaHbX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790651852476208-21193-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790651852476208-21193-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790651852476208-21193-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790651852476208-21193-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.178.126:34731
--webserver_interface=127.20.178.126
--webserver_port=0
--builtin_ntp_servers=127.20.178.84:34847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.178.126:34731
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260929 03:18:26.219201 30899 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:18:26.219394 30899 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:18:26.219415 30899 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:18:26.220572 30899 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:18:26.220613 30899 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:18:26.220624 30899 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:18:26.220661 30899 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:18:26.220674 30899 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:18:26.221971 30899 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.178.84:34847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790651852476208-21193-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790651852476208-21193-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.178.126:34731
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.178.126:34731
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790651852476208-21193-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.178.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30899
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790651852476208-21193-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:18:26.222236 30899 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:18:26.222512 30899 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:18:26.227070 30904 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:18:26.227070 30907 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:18:26.227077 30905 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:18:26.227418 30899 server_base.cc:1034] running on GCE node
I20260929 03:18:26.227700 30899 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:18:26.228000 30899 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:18:26.229144 30899 hybrid_clock.cc:648] HybridClock initialized: now 1790651906229121 us; error 37 us; skew 500 ppm
I20260929 03:18:26.230294 30899 webserver.cc:460] Webserver started at http://127.20.178.126:34907/ using document root <none> and password file <none>
I20260929 03:18:26.230512 30899 fs_manager.cc:359] Metadata directory not provided
I20260929 03:18:26.230556 30899 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:18:26.230697 30899 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:18:26.231689 30899 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790651852476208-21193-0/minicluster-data/master-0/data/instance:
uuid: "22df414040a34c28a367d94fd753c6d5"
format_stamp: "Formatted at 2026-09-29 03:18:26 on dist-test-slave-c9qh"
server_key: "929467ee52228b6a0e586d65e8af0a65"
server_key_iv: "3c7b46001fd8a6dd97eaef0dfeca6401"
server_key_version: "encryptionkey@0"
I20260929 03:18:26.232020 30899 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790651852476208-21193-0/minicluster-data/master-0/wal/instance:
uuid: "22df414040a34c28a367d94fd753c6d5"
format_stamp: "Formatted at 2026-09-29 03:18:26 on dist-test-slave-c9qh"
server_key: "929467ee52228b6a0e586d65e8af0a65"
server_key_iv: "3c7b46001fd8a6dd97eaef0dfeca6401"
server_key_version: "encryptionkey@0"
I20260929 03:18:26.233325 30899 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 03:18:26.233994 30913 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:18:26.234179 30899 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 03:18:26.234272 30899 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790651852476208-21193-0/minicluster-data/master-0/data,/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790651852476208-21193-0/minicluster-data/master-0/wal
uuid: "22df414040a34c28a367d94fd753c6d5"
format_stamp: "Formatted at 2026-09-29 03:18:26 on dist-test-slave-c9qh"
server_key: "929467ee52228b6a0e586d65e8af0a65"
server_key_iv: "3c7b46001fd8a6dd97eaef0dfeca6401"
server_key_version: "encryptionkey@0"
I20260929 03:18:26.234335 30899 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790651852476208-21193-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790651852476208-21193-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790651852476208-21193-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:18:26.237528 30899 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:18:26.237788 30899 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:18:26.237917 30899 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:18:26.241670 30899 rpc_server.cc:307] RPC server started. Bound to: 127.20.178.126:34731
I20260929 03:18:26.241698 30965 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.178.126:34731 every 8 connection(s)
I20260929 03:18:26.242017 30899 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790651852476208-21193-0/minicluster-data/master-0/data/info.pb
I20260929 03:18:26.242606 30966 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:18:26.244179 21193 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuoaHbX/build/release/bin/kudu as pid 30899
I20260929 03:18:26.244282 21193 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790651852476208-21193-0/minicluster-data/master-0/wal/instance
I20260929 03:18:26.244493 21193 external_mini_cluster.cc:1428] Setting key b8be4dc47808a1402472474fc285204f
I20260929 03:18:26.245651 30966 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22df414040a34c28a367d94fd753c6d5: Bootstrap starting.
I20260929 03:18:26.246246 30966 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 22df414040a34c28a367d94fd753c6d5: Neither blocks nor log segments found. Creating new log.
I20260929 03:18:26.246572 30966 log.cc:824] T 00000000000000000000000000000000 P 22df414040a34c28a367d94fd753c6d5: Log is configured to *not* fsync() on all Append() calls
I20260929 03:18:26.247275 30969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:26.245048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40090 (local address 127.20.178.126:34731)
0929 03:18:26.245226 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0929 03:18:26.245235 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 03:18:26.245314 (+ 79us) server_negotiation.cc:407] Connection header received
0929 03:18:26.245822 (+ 508us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:18:26.245829 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:18:26.245868 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:18:26.246005 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:18:26.246691 (+ 686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:18:26.246695 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:18:26.246695 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:18:26.246699 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:18:26.246748 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:18:26.246758 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:18:26.246845 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0929 03:18:26.247044 (+ 199us) server_negotiation.cc:299] Negotiation successful
0929 03:18:26.247103 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":77,"threads_started":1}
I20260929 03:18:26.247349 30966 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22df414040a34c28a367d94fd753c6d5: No bootstrap required, opened a new log
I20260929 03:18:26.248690 30966 raft_consensus.cc:348] T 00000000000000000000000000000000 P 22df414040a34c28a367d94fd753c6d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22df414040a34c28a367d94fd753c6d5" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 34731 } }
I20260929 03:18:26.248847 30966 raft_consensus.cc:374] T 00000000000000000000000000000000 P 22df414040a34c28a367d94fd753c6d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:18:26.248905 30966 raft_consensus.cc:729] T 00000000000000000000000000000000 P 22df414040a34c28a367d94fd753c6d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22df414040a34c28a367d94fd753c6d5, State: Initialized, Role: FOLLOWER
I20260929 03:18:26.249040 30966 consensus_queue.cc:260] T 00000000000000000000000000000000 P 22df414040a34c28a367d94fd753c6d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22df414040a34c28a367d94fd753c6d5" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 34731 } }
I20260929 03:18:26.249115 30966 raft_consensus.cc:388] T 00000000000000000000000000000000 P 22df414040a34c28a367d94fd753c6d5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:18:26.249156 30966 raft_consensus.cc:482] T 00000000000000000000000000000000 P 22df414040a34c28a367d94fd753c6d5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:18:26.249202 30966 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 22df414040a34c28a367d94fd753c6d5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:18:26.249917 30966 raft_consensus.cc:504] T 00000000000000000000000000000000 P 22df414040a34c28a367d94fd753c6d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22df414040a34c28a367d94fd753c6d5" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 34731 } }
I20260929 03:18:26.250027 30966 leader_election.cc:302] T 00000000000000000000000000000000 P 22df414040a34c28a367d94fd753c6d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 22df414040a34c28a367d94fd753c6d5; no voters:
I20260929 03:18:26.250188 30966 leader_election.cc:288] T 00000000000000000000000000000000 P 22df414040a34c28a367d94fd753c6d5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:18:26.250238 30971 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 22df414040a34c28a367d94fd753c6d5 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:18:26.250376 30971 raft_consensus.cc:686] T 00000000000000000000000000000000 P 22df414040a34c28a367d94fd753c6d5 [term 1 LEADER]: Becoming Leader. State: Replica: 22df414040a34c28a367d94fd753c6d5, State: Running, Role: LEADER
I20260929 03:18:26.250430 30966 sys_catalog.cc:564] T 00000000000000000000000000000000 P 22df414040a34c28a367d94fd753c6d5 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:18:26.250464 30971 consensus_queue.cc:237] T 00000000000000000000000000000000 P 22df414040a34c28a367d94fd753c6d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22df414040a34c28a367d94fd753c6d5" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 34731 } }
I20260929 03:18:26.250793 30973 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22df414040a34c28a367d94fd753c6d5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 22df414040a34c28a367d94fd753c6d5. Latest consensus state: current_term: 1 leader_uuid: "22df414040a34c28a367d94fd753c6d5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22df414040a34c28a367d94fd753c6d5" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 34731 } } }
I20260929 03:18:26.250808 30972 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22df414040a34c28a367d94fd753c6d5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "22df414040a34c28a367d94fd753c6d5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22df414040a34c28a367d94fd753c6d5" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 34731 } } }
I20260929 03:18:26.250919 30972 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22df414040a34c28a367d94fd753c6d5 [sys.catalog]: This master's current role is: LEADER
I20260929 03:18:26.251080 30973 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22df414040a34c28a367d94fd753c6d5 [sys.catalog]: This master's current role is: LEADER
I20260929 03:18:26.251142 30978 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:18:26.251588 30978 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:18:26.257500 30978 catalog_manager.cc:1347] Generated new cluster ID: 693a434f467049b6bc64b90c7b645be8
I20260929 03:18:26.257572 30978 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:18:26.262305 30978 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:18:26.262465 30978 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:18:26.266047 30978 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 22df414040a34c28a367d94fd753c6d5: Generated new TSK 0
I20260929 03:18:26.266222 30978 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 03:18:26.267848 21193 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuoaHbX/build/release/bin/kudu
/tmp/dist-test-taskuoaHbX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790651852476208-21193-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790651852476208-21193-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790651852476208-21193-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790651852476208-21193-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.178.65:0
--local_ip_for_outbound_sockets=127.20.178.65
--webserver_interface=127.20.178.65
--webserver_port=0
--tserver_master_addrs=127.20.178.126:34731
--builtin_ntp_servers=127.20.178.84:34847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260929 03:18:26.355731 30990 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:18:26.355968 30990 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:18:26.356009 30990 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:18:26.357277 30990 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:18:26.357339 30990 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:18:26.357383 30990 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.178.65
I20260929 03:18:26.359251 30990 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.178.84:34847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790651852476208-21193-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790651852476208-21193-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.178.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790651852476208-21193-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.178.65
--webserver_port=0
--tserver_master_addrs=127.20.178.126:34731
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30990
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.178.65
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790651852476208-21193-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:18:26.359524 30990 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:18:26.359819 30990 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:18:26.365087 30996 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:18:26.365104 30998 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:18:26.365231 30995 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:18:26.365284 30990 server_base.cc:1034] running on GCE node
I20260929 03:18:26.365479 30990 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:18:26.365717 30990 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:18:26.366873 30990 hybrid_clock.cc:648] HybridClock initialized: now 1790651906366854 us; error 34 us; skew 500 ppm
I20260929 03:18:26.368335 30990 webserver.cc:460] Webserver started at http://127.20.178.65:37083/ using document root <none> and password file <none>
I20260929 03:18:26.368607 30990 fs_manager.cc:359] Metadata directory not provided
I20260929 03:18:26.368708 30990 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:18:26.368870 30990 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:18:26.370030 30990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790651852476208-21193-0/minicluster-data/ts-0/data/instance:
uuid: "3ec97d7ee6aa492ea3856f2258d7b12e"
format_stamp: "Formatted at 2026-09-29 03:18:26 on dist-test-slave-c9qh"
server_key: "6968aad013a970cfa16a6998391cc9f9"
server_key_iv: "a8e610fb6d6158462eb25a7a10142d08"
server_key_version: "encryptionkey@0"
I20260929 03:18:26.370401 30990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790651852476208-21193-0/minicluster-data/ts-0/wal/instance:
uuid: "3ec97d7ee6aa492ea3856f2258d7b12e"
format_stamp: "Formatted at 2026-09-29 03:18:26 on dist-test-slave-c9qh"
server_key: "6968aad013a970cfa16a6998391cc9f9"
server_key_iv: "a8e610fb6d6158462eb25a7a10142d08"
server_key_version: "encryptionkey@0"
I20260929 03:18:26.371734 30990 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 03:18:26.372478 31004 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:18:26.372655 30990 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 03:18:26.372742 30990 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790651852476208-21193-0/minicluster-data/ts-0/data,/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790651852476208-21193-0/minicluster-data/ts-0/wal
uuid: "3ec97d7ee6aa492ea3856f2258d7b12e"
format_stamp: "Formatted at 2026-09-29 03:18:26 on dist-test-slave-c9qh"
server_key: "6968aad013a970cfa16a6998391cc9f9"
server_key_iv: "a8e610fb6d6158462eb25a7a10142d08"
server_key_version: "encryptionkey@0"
I20260929 03:18:26.372792 30990 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790651852476208-21193-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790651852476208-21193-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790651852476208-21193-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:18:26.375967 30990 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:18:26.376230 30990 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:18:26.376371 30990 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:18:26.376576 30990 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:18:26.376889 30990 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:18:26.376942 30990 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:18:26.376981 30990 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:18:26.377009 30990 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:18:26.382800 30990 rpc_server.cc:307] RPC server started. Bound to: 127.20.178.65:33375
I20260929 03:18:26.382843 31117 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.178.65:33375 every 8 connection(s)
I20260929 03:18:26.383136 30990 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790651852476208-21193-0/minicluster-data/ts-0/data/info.pb
I20260929 03:18:26.385928 30969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:26.383864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.65:60147 (local address 127.20.178.126:34731)
0929 03:18:26.383928 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0929 03:18:26.383930 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:18:26.384142 (+ 212us) server_negotiation.cc:407] Connection header received
0929 03:18:26.384567 (+ 425us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:18:26.384568 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:18:26.384573 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:18:26.384663 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:18:26.385750 (+ 1087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:18:26.385751 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:18:26.385752 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:18:26.385755 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:18:26.385768 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:18:26.385773 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:18:26.385815 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0929 03:18:26.385867 (+ 52us) server_negotiation.cc:299] Negotiation successful
0929 03:18:26.385878 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260929 03:18:26.386140 31120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:26.383852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:34731 (local address 127.20.178.65:60147)
0929 03:18:26.384016 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:26.384056 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 03:18:26.384487 (+ 431us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:18:26.384739 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:18:26.384742 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:18:26.384749 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:18:26.385598 (+ 849us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 03:18:26.385599 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:18:26.385625 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 03:18:26.385651 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:18:26.385819 (+ 168us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:18:26.385822 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:18:26.385824 (+ 2us) client_negotiation.cc:769] Sending connection context
0929 03:18:26.385873 (+ 49us) client_negotiation.cc:240] Negotiation successful
0929 03:18:26.385917 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":75,"threads_started":1}
I20260929 03:18:26.386596 31118 heartbeater.cc:347] Connected to a master server at 127.20.178.126:34731
I20260929 03:18:26.386673 31118 heartbeater.cc:464] Registering TS with master...
I20260929 03:18:26.386828 31118 heartbeater.cc:511] Master 127.20.178.126:34731 requested a full tablet report, sending...
I20260929 03:18:26.387302 30930 ts_manager.cc:194] Registered new tserver with Master: 3ec97d7ee6aa492ea3856f2258d7b12e (127.20.178.65:33375)
I20260929 03:18:26.389501 21193 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuoaHbX/build/release/bin/kudu as pid 30990
I20260929 03:18:26.389564 21193 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790651852476208-21193-0/minicluster-data/ts-0/wal/instance
I20260929 03:18:26.389711 21193 external_mini_cluster.cc:1428] Setting key 434280fa39835ae58b4043b21336e3d3
I20260929 03:18:26.390728 21193 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuoaHbX/build/release/bin/kudu
/tmp/dist-test-taskuoaHbX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790651852476208-21193-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790651852476208-21193-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790651852476208-21193-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790651852476208-21193-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.178.66:0
--local_ip_for_outbound_sockets=127.20.178.66
--webserver_interface=127.20.178.66
--webserver_port=0
--tserver_master_addrs=127.20.178.126:34731
--builtin_ntp_servers=127.20.178.84:34847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260929 03:18:26.469291 31121 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:18:26.469493 31121 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:18:26.469515 31121 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:18:26.470587 31121 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:18:26.470618 31121 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:18:26.470642 31121 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.178.66
I20260929 03:18:26.471879 31121 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.178.84:34847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790651852476208-21193-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790651852476208-21193-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.178.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790651852476208-21193-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.178.66
--webserver_port=0
--tserver_master_addrs=127.20.178.126:34731
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31121
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.178.66
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790651852476208-21193-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:18:26.472054 31121 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:18:26.472276 31121 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:18:26.477154 31127 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:18:26.477187 31126 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:18:26.477187 31129 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:18:26.477587 31121 server_base.cc:1034] running on GCE node
I20260929 03:18:26.477766 31121 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:18:26.477958 31121 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:18:26.479099 31121 hybrid_clock.cc:648] HybridClock initialized: now 1790651906479082 us; error 27 us; skew 500 ppm
I20260929 03:18:26.486057 31121 webserver.cc:460] Webserver started at http://127.20.178.66:35275/ using document root <none> and password file <none>
I20260929 03:18:26.486335 31121 fs_manager.cc:359] Metadata directory not provided
I20260929 03:18:26.486388 31121 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:18:26.486524 31121 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:18:26.487569 31121 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790651852476208-21193-0/minicluster-data/ts-1/data/instance:
uuid: "e86e8c033ac54483ab83481073f294ce"
format_stamp: "Formatted at 2026-09-29 03:18:26 on dist-test-slave-c9qh"
server_key: "17b8a13c726c86dc0f9f069101ea8b7a"
server_key_iv: "17920d65e5fad90f47c56ce6f1d52b59"
server_key_version: "encryptionkey@0"
I20260929 03:18:26.487919 31121 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790651852476208-21193-0/minicluster-data/ts-1/wal/instance:
uuid: "e86e8c033ac54483ab83481073f294ce"
format_stamp: "Formatted at 2026-09-29 03:18:26 on dist-test-slave-c9qh"
server_key: "17b8a13c726c86dc0f9f069101ea8b7a"
server_key_iv: "17920d65e5fad90f47c56ce6f1d52b59"
server_key_version: "encryptionkey@0"
I20260929 03:18:26.489275 31121 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 03:18:26.489976 31135 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:18:26.490142 31121 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 03:18:26.490257 31121 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790651852476208-21193-0/minicluster-data/ts-1/data,/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790651852476208-21193-0/minicluster-data/ts-1/wal
uuid: "e86e8c033ac54483ab83481073f294ce"
format_stamp: "Formatted at 2026-09-29 03:18:26 on dist-test-slave-c9qh"
server_key: "17b8a13c726c86dc0f9f069101ea8b7a"
server_key_iv: "17920d65e5fad90f47c56ce6f1d52b59"
server_key_version: "encryptionkey@0"
I20260929 03:18:26.490331 31121 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790651852476208-21193-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790651852476208-21193-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790651852476208-21193-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:18:26.493711 31121 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:18:26.493965 31121 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:18:26.494107 31121 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:18:26.494311 31121 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:18:26.494612 31121 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:18:26.494664 31121 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:18:26.494702 31121 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:18:26.494730 31121 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:18:26.500389 31121 rpc_server.cc:307] RPC server started. Bound to: 127.20.178.66:42999
I20260929 03:18:26.500432 31248 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.178.66:42999 every 8 connection(s)
I20260929 03:18:26.500790 31121 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790651852476208-21193-0/minicluster-data/ts-1/data/info.pb
I20260929 03:18:26.503436 30969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:26.501453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.66:59879 (local address 127.20.178.126:34731)
0929 03:18:26.501524 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0929 03:18:26.501525 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 03:18:26.501756 (+ 231us) server_negotiation.cc:407] Connection header received
0929 03:18:26.502156 (+ 400us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:18:26.502157 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:18:26.502161 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:18:26.502219 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:18:26.503254 (+ 1035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:18:26.503255 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:18:26.503255 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:18:26.503259 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:18:26.503272 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:18:26.503276 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:18:26.503326 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0929 03:18:26.503374 (+ 48us) server_negotiation.cc:299] Negotiation successful
0929 03:18:26.503385 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 03:18:26.503618 31251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:26.501438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:34731 (local address 127.20.178.66:59879)
0929 03:18:26.501605 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:26.501642 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 03:18:26.502064 (+ 422us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:18:26.502266 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:18:26.502270 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:18:26.502277 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:18:26.503110 (+ 833us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 03:18:26.503111 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:18:26.503136 (+ 25us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 03:18:26.503163 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:18:26.503327 (+ 164us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:18:26.503329 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:18:26.503331 (+ 2us) client_negotiation.cc:769] Sending connection context
0929 03:18:26.503382 (+ 51us) client_negotiation.cc:240] Negotiation successful
0929 03:18:26.503426 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":78,"threads_started":1}
I20260929 03:18:26.504069 31249 heartbeater.cc:347] Connected to a master server at 127.20.178.126:34731
I20260929 03:18:26.504141 31249 heartbeater.cc:464] Registering TS with master...
I20260929 03:18:26.504307 31249 heartbeater.cc:511] Master 127.20.178.126:34731 requested a full tablet report, sending...
I20260929 03:18:26.504735 30930 ts_manager.cc:194] Registered new tserver with Master: e86e8c033ac54483ab83481073f294ce (127.20.178.66:42999)
I20260929 03:18:26.505369 21193 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuoaHbX/build/release/bin/kudu as pid 31121
I20260929 03:18:26.505442 21193 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790651852476208-21193-0/minicluster-data/ts-1/wal/instance
I20260929 03:18:26.505636 21193 external_mini_cluster.cc:1428] Setting key 3d928b165846acf625b52cbb2bc0a150
I20260929 03:18:26.506466 21193 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuoaHbX/build/release/bin/kudu
/tmp/dist-test-taskuoaHbX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790651852476208-21193-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790651852476208-21193-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790651852476208-21193-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790651852476208-21193-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.178.67:0
--local_ip_for_outbound_sockets=127.20.178.67
--webserver_interface=127.20.178.67
--webserver_port=0
--tserver_master_addrs=127.20.178.126:34731
--builtin_ntp_servers=127.20.178.84:34847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260929 03:18:26.603870 31252 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:18:26.604063 31252 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:18:26.604084 31252 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:18:26.605180 31252 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:18:26.605211 31252 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:18:26.605242 31252 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.178.67
I20260929 03:18:26.606472 31252 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.178.84:34847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790651852476208-21193-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790651852476208-21193-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.178.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790651852476208-21193-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.178.67
--webserver_port=0
--tserver_master_addrs=127.20.178.126:34731
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31252
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.178.67
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790651852476208-21193-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:18:26.606645 31252 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:18:26.606863 31252 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:18:26.611723 31257 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:18:26.611740 31252 server_base.cc:1034] running on GCE node
W20260929 03:18:26.611711 31258 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:18:26.611691 31260 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:18:26.612159 31252 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:18:26.612402 31252 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:18:26.613597 31252 hybrid_clock.cc:648] HybridClock initialized: now 1790651906613586 us; error 41 us; skew 500 ppm
I20260929 03:18:26.614840 31252 webserver.cc:460] Webserver started at http://127.20.178.67:43217/ using document root <none> and password file <none>
I20260929 03:18:26.615084 31252 fs_manager.cc:359] Metadata directory not provided
I20260929 03:18:26.615151 31252 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:18:26.615289 31252 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:18:26.616350 31252 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790651852476208-21193-0/minicluster-data/ts-2/data/instance:
uuid: "29af698caf3f48f9979939942b3fe228"
format_stamp: "Formatted at 2026-09-29 03:18:26 on dist-test-slave-c9qh"
server_key: "b75766f9ee68502bbccc31cd8a58403e"
server_key_iv: "ffd5077e6afe79c4d7e55e5269417b00"
server_key_version: "encryptionkey@0"
I20260929 03:18:26.616773 31252 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790651852476208-21193-0/minicluster-data/ts-2/wal/instance:
uuid: "29af698caf3f48f9979939942b3fe228"
format_stamp: "Formatted at 2026-09-29 03:18:26 on dist-test-slave-c9qh"
server_key: "b75766f9ee68502bbccc31cd8a58403e"
server_key_iv: "ffd5077e6afe79c4d7e55e5269417b00"
server_key_version: "encryptionkey@0"
I20260929 03:18:26.618040 31252 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 03:18:26.618713 31266 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:18:26.618878 31252 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 03:18:26.618973 31252 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790651852476208-21193-0/minicluster-data/ts-2/data,/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790651852476208-21193-0/minicluster-data/ts-2/wal
uuid: "29af698caf3f48f9979939942b3fe228"
format_stamp: "Formatted at 2026-09-29 03:18:26 on dist-test-slave-c9qh"
server_key: "b75766f9ee68502bbccc31cd8a58403e"
server_key_iv: "ffd5077e6afe79c4d7e55e5269417b00"
server_key_version: "encryptionkey@0"
I20260929 03:18:26.619059 31252 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790651852476208-21193-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790651852476208-21193-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790651852476208-21193-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:18:26.622327 31252 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:18:26.622574 31252 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:18:26.622735 31252 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:18:26.622934 31252 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:18:26.623221 31252 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:18:26.623275 31252 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:18:26.623314 31252 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:18:26.623340 31252 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:18:26.628883 31252 rpc_server.cc:307] RPC server started. Bound to: 127.20.178.67:40251
I20260929 03:18:26.628925 31379 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.178.67:40251 every 8 connection(s)
I20260929 03:18:26.629227 31252 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790651852476208-21193-0/minicluster-data/ts-2/data/info.pb
I20260929 03:18:26.630939 21193 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuoaHbX/build/release/bin/kudu as pid 31252
I20260929 03:18:26.631029 21193 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790651852476208-21193-0/minicluster-data/ts-2/wal/instance
I20260929 03:18:26.631202 21193 external_mini_cluster.cc:1428] Setting key 9d7d4cd3c4427a0196e61be7a0726a14
I20260929 03:18:26.632273 30969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:26.629915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.67:57033 (local address 127.20.178.126:34731)
0929 03:18:26.629976 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0929 03:18:26.629977 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 03:18:26.630177 (+ 200us) server_negotiation.cc:407] Connection header received
0929 03:18:26.630599 (+ 422us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:18:26.630599 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:18:26.630603 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:18:26.630663 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:18:26.632007 (+ 1344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:18:26.632008 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:18:26.632008 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:18:26.632012 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:18:26.632029 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:18:26.632034 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:18:26.632093 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0929 03:18:26.632189 (+ 96us) server_negotiation.cc:299] Negotiation successful
0929 03:18:26.632203 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260929 03:18:26.632488 31382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:26.629892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:34731 (local address 127.20.178.67:57033)
0929 03:18:26.630055 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:26.630092 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 03:18:26.630523 (+ 431us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:18:26.630720 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:18:26.630723 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:18:26.630730 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:18:26.631878 (+ 1148us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 03:18:26.631879 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:18:26.631913 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 03:18:26.631949 (+ 36us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:18:26.632102 (+ 153us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:18:26.632106 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:18:26.632109 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 03:18:26.632183 (+ 74us) client_negotiation.cc:240] Negotiation successful
0929 03:18:26.632261 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":74,"threads_started":1}
I20260929 03:18:26.632922 31380 heartbeater.cc:347] Connected to a master server at 127.20.178.126:34731
I20260929 03:18:26.633021 31380 heartbeater.cc:464] Registering TS with master...
I20260929 03:18:26.633159 31380 heartbeater.cc:511] Master 127.20.178.126:34731 requested a full tablet report, sending...
I20260929 03:18:26.633494 30930 ts_manager.cc:194] Registered new tserver with Master: 29af698caf3f48f9979939942b3fe228 (127.20.178.67:40251)
I20260929 03:18:26.642978 21193 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260929 03:18:26.644698 30969 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:18:26.644243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40092 (local address 127.20.178.126:34731)
0929 03:18:26.644300 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0929 03:18:26.644302 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:18:26.644414 (+ 112us) server_negotiation.cc:407] Connection header received
0929 03:18:26.644439 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:18:26.644439 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:18:26.644443 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:18:26.644489 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:18:26.644614 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40092: BlockingRecv error: recv got EOF from 127.0.0.1:40092 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
I20260929 03:18:26.644739 21193 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuoaHbX/build/release/bin/kudu with pid 30990
I20260929 03:18:26.649801 21193 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuoaHbX/build/release/bin/kudu with pid 31121
I20260929 03:18:26.654152 21193 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuoaHbX/build/release/bin/kudu with pid 31252
I20260929 03:18:26.658318 21193 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuoaHbX/build/release/bin/kudu with pid 30899
2026-09-29T03:18:26Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (543 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651852476208-21193-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:18:26 dist-test-slave-c9qh krb5kdc[31390](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:18:26 dist-test-slave-c9qh krb5kdc[31390](info): set up 2 sockets
Sep 29 03:18:26 dist-test-slave-c9qh krb5kdc[31390](info): commencing operation
krb5kdc: starting...
W20260929 03:18:27.693547 21193 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651852476208-21193-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651852476208-21193-0/krb5kdc/test-user.keytab.
Sep 29 03:18:27 dist-test-slave-c9qh krb5kdc[31390](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651907, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:18:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:18:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.178.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.178.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.178.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.126.keytab.
Entry for principal kudu/127.20.178.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.126.keytab.
I20260929 03:18:27.811622 21193 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuoaHbX/build/release/bin/kudu
/tmp/dist-test-taskuoaHbX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651852476208-21193-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651852476208-21193-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651852476208-21193-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651852476208-21193-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.178.126:41707
--webserver_interface=127.20.178.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.126.keytab
--principal=kudu/127.20.178.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.178.84:46455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.178.126:41707
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651852476208-21193-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651852476208-21193-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651852476208-21193-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:18:27.891099 31403 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:18:27.891295 31403 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:18:27.891317 31403 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:18:27.892475 31403 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:18:27.892517 31403 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:18:27.892532 31403 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:18:27.892545 31403 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:18:27.892558 31403 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:18:27.894070 31403 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.178.84:46455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651852476208-21193-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651852476208-21193-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.178.126:41707
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.126.keytab
--principal=kudu/127.20.178.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.178.126:41707
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651852476208-21193-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.178.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31403
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651852476208-21193-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:18:27.894315 31403 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:18:27.894549 31403 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:18:27.899271 31408 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:18:27.899356 31411 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:18:27.899338 31409 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:18:27.899489 31403 server_base.cc:1034] running on GCE node
I20260929 03:18:27.899792 31403 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:18:27.900080 31403 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:18:27.901394 31403 hybrid_clock.cc:648] HybridClock initialized: now 1790651907901372 us; error 54 us; skew 500 ppm
Sep 29 03:18:27 dist-test-slave-c9qh krb5kdc[31390](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651907, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:18:27.903749 31403 init.cc:377] Logged in from keytab as kudu/127.20.178.126@KRBTEST.COM (short username kudu)
I20260929 03:18:27.904178 31403 webserver.cc:460] Webserver started at http://127.20.178.126:35693/ using document root <none> and password file <none>
I20260929 03:18:27.904399 31403 fs_manager.cc:359] Metadata directory not provided
I20260929 03:18:27.904464 31403 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:18:27.904611 31403 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:18:27.905730 31403 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651852476208-21193-0/minicluster-data/master-0/data/instance:
uuid: "045d51f9ab9e4873a97eacfd153c3919"
format_stamp: "Formatted at 2026-09-29 03:18:27 on dist-test-slave-c9qh"
server_key: "9f472f5c5217c789ec12bd7ee87154b4"
server_key_iv: "f57e4fcb8d1052247de0b7f67439359d"
server_key_version: "encryptionkey@0"
I20260929 03:18:27.906102 31403 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651852476208-21193-0/minicluster-data/master-0/wal/instance:
uuid: "045d51f9ab9e4873a97eacfd153c3919"
format_stamp: "Formatted at 2026-09-29 03:18:27 on dist-test-slave-c9qh"
server_key: "9f472f5c5217c789ec12bd7ee87154b4"
server_key_iv: "f57e4fcb8d1052247de0b7f67439359d"
server_key_version: "encryptionkey@0"
I20260929 03:18:27.907392 31403 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260929 03:18:27.908062 31418 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:18:27.908252 31403 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 03:18:27.908353 31403 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651852476208-21193-0/minicluster-data/master-0/data,/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651852476208-21193-0/minicluster-data/master-0/wal
uuid: "045d51f9ab9e4873a97eacfd153c3919"
format_stamp: "Formatted at 2026-09-29 03:18:27 on dist-test-slave-c9qh"
server_key: "9f472f5c5217c789ec12bd7ee87154b4"
server_key_iv: "f57e4fcb8d1052247de0b7f67439359d"
server_key_version: "encryptionkey@0"
I20260929 03:18:27.908423 31403 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651852476208-21193-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651852476208-21193-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651852476208-21193-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:18:27.915378 31403 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:18:27.917271 31403 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:18:27.917423 31403 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:18:27.921104 31403 rpc_server.cc:307] RPC server started. Bound to: 127.20.178.126:41707
I20260929 03:18:27.921134 31470 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.178.126:41707 every 8 connection(s)
I20260929 03:18:27.921466 31403 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651852476208-21193-0/minicluster-data/master-0/data/info.pb
I20260929 03:18:27.922432 31471 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:18:27.924590 31471 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 045d51f9ab9e4873a97eacfd153c3919: Bootstrap starting.
I20260929 03:18:27.925208 31471 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 045d51f9ab9e4873a97eacfd153c3919: Neither blocks nor log segments found. Creating new log.
I20260929 03:18:27.925532 31471 log.cc:824] T 00000000000000000000000000000000 P 045d51f9ab9e4873a97eacfd153c3919: Log is configured to *not* fsync() on all Append() calls
I20260929 03:18:27.926081 21193 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuoaHbX/build/release/bin/kudu as pid 31403
I20260929 03:18:27.926156 21193 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651852476208-21193-0/minicluster-data/master-0/wal/instance
I20260929 03:18:27.926321 31471 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 045d51f9ab9e4873a97eacfd153c3919: No bootstrap required, opened a new log
I20260929 03:18:27.926366 21193 external_mini_cluster.cc:1428] Setting key b56d0576783deda3c6389754c25b7e9e
I20260929 03:18:27.928596 31471 raft_consensus.cc:348] T 00000000000000000000000000000000 P 045d51f9ab9e4873a97eacfd153c3919 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "045d51f9ab9e4873a97eacfd153c3919" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 41707 } }
I20260929 03:18:27.928790 31471 raft_consensus.cc:374] T 00000000000000000000000000000000 P 045d51f9ab9e4873a97eacfd153c3919 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:18:27.928833 31471 raft_consensus.cc:729] T 00000000000000000000000000000000 P 045d51f9ab9e4873a97eacfd153c3919 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 045d51f9ab9e4873a97eacfd153c3919, State: Initialized, Role: FOLLOWER
I20260929 03:18:27.928974 31471 consensus_queue.cc:260] T 00000000000000000000000000000000 P 045d51f9ab9e4873a97eacfd153c3919 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "045d51f9ab9e4873a97eacfd153c3919" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 41707 } }
I20260929 03:18:27.929049 31471 raft_consensus.cc:388] T 00000000000000000000000000000000 P 045d51f9ab9e4873a97eacfd153c3919 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:18:27.929077 31471 raft_consensus.cc:482] T 00000000000000000000000000000000 P 045d51f9ab9e4873a97eacfd153c3919 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:18:27.929131 31471 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 045d51f9ab9e4873a97eacfd153c3919 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:18:27.929782 31471 raft_consensus.cc:504] T 00000000000000000000000000000000 P 045d51f9ab9e4873a97eacfd153c3919 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "045d51f9ab9e4873a97eacfd153c3919" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 41707 } }
I20260929 03:18:27.929884 31471 leader_election.cc:302] T 00000000000000000000000000000000 P 045d51f9ab9e4873a97eacfd153c3919 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 045d51f9ab9e4873a97eacfd153c3919; no voters:
I20260929 03:18:27.930025 31471 leader_election.cc:288] T 00000000000000000000000000000000 P 045d51f9ab9e4873a97eacfd153c3919 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:18:27.930086 31476 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 045d51f9ab9e4873a97eacfd153c3919 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:18:27.930264 31471 sys_catalog.cc:564] T 00000000000000000000000000000000 P 045d51f9ab9e4873a97eacfd153c3919 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:18:27.930285 31476 raft_consensus.cc:686] T 00000000000000000000000000000000 P 045d51f9ab9e4873a97eacfd153c3919 [term 1 LEADER]: Becoming Leader. State: Replica: 045d51f9ab9e4873a97eacfd153c3919, State: Running, Role: LEADER
I20260929 03:18:27.930377 31476 consensus_queue.cc:237] T 00000000000000000000000000000000 P 045d51f9ab9e4873a97eacfd153c3919 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "045d51f9ab9e4873a97eacfd153c3919" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 41707 } }
I20260929 03:18:27.930716 31476 sys_catalog.cc:455] T 00000000000000000000000000000000 P 045d51f9ab9e4873a97eacfd153c3919 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 045d51f9ab9e4873a97eacfd153c3919. Latest consensus state: current_term: 1 leader_uuid: "045d51f9ab9e4873a97eacfd153c3919" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "045d51f9ab9e4873a97eacfd153c3919" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 41707 } } }
I20260929 03:18:27.930727 31477 sys_catalog.cc:455] T 00000000000000000000000000000000 P 045d51f9ab9e4873a97eacfd153c3919 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "045d51f9ab9e4873a97eacfd153c3919" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "045d51f9ab9e4873a97eacfd153c3919" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 41707 } } }
I20260929 03:18:27.930931 31477 sys_catalog.cc:458] T 00000000000000000000000000000000 P 045d51f9ab9e4873a97eacfd153c3919 [sys.catalog]: This master's current role is: LEADER
I20260929 03:18:27.931110 31476 sys_catalog.cc:458] T 00000000000000000000000000000000 P 045d51f9ab9e4873a97eacfd153c3919 [sys.catalog]: This master's current role is: LEADER
I20260929 03:18:27.931344 31485 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:18:27.931890 31485 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:18:27.933405 31485 catalog_manager.cc:1347] Generated new cluster ID: 8485953f856e4d44ab1efe726bc4bd16
I20260929 03:18:27.933460 31485 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 29 03:18:27 dist-test-slave-c9qh krb5kdc[31390](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651907, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.178.126@KRBTEST.COM
I20260929 03:18:27.938055 31475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:27.926939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41338 (local address 127.20.178.126:41707)
0929 03:18:27.927136 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0929 03:18:27.927145 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 03:18:27.927176 (+ 31us) server_negotiation.cc:407] Connection header received
0929 03:18:27.927771 (+ 595us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:18:27.927780 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:18:27.927841 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:18:27.928000 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:18:27.928557 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:27.930215 (+ 1658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:27.931435 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:27.932100 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:27.932231 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:27.934737 (+ 2506us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:18:27.934741 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:18:27.934741 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:18:27.934752 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:18:27.936373 (+ 1621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:18:27.936788 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:18:27.936791 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:18:27.936791 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:18:27.936880 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:18:27.937080 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:18:27.937081 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:18:27.937082 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:18:27.937308 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:18:27.937465 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0929 03:18:27.937572 (+ 107us) server_negotiation.cc:299] Negotiation successful
0929 03:18:27.937808 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":102,"threads_started":1}
I20260929 03:18:27.940135 31485 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:18:27.940853 31485 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:18:27.943791 31485 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 045d51f9ab9e4873a97eacfd153c3919: Generated new TSK 0
I20260929 03:18:27.943945 31485 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.178.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.178.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.178.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.65.keytab.
Entry for principal kudu/127.20.178.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.65.keytab.
I20260929 03:18:27.969547 21193 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuoaHbX/build/release/bin/kudu
/tmp/dist-test-taskuoaHbX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651852476208-21193-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651852476208-21193-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651852476208-21193-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651852476208-21193-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.178.65:0
--local_ip_for_outbound_sockets=127.20.178.65
--webserver_interface=127.20.178.65
--webserver_port=0
--tserver_master_addrs=127.20.178.126:41707
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.65.keytab
--principal=kudu/127.20.178.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.178.84:46455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651852476208-21193-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651852476208-21193-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651852476208-21193-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:18:28.051878 31497 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:18:28.052103 31497 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:18:28.052146 31497 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:18:28.053345 31497 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:18:28.053412 31497 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:18:28.053459 31497 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.178.65
I20260929 03:18:28.054908 31497 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.178.84:46455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651852476208-21193-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651852476208-21193-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.65.keytab
--principal=kudu/127.20.178.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.178.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651852476208-21193-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.178.65
--webserver_port=0
--tserver_master_addrs=127.20.178.126:41707
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31497
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.178.65
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651852476208-21193-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:18:28.055166 31497 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:18:28.055404 31497 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:18:28.061137 31502 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:18:28.061273 31503 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:18:28.061280 31505 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:18:28.061355 31497 server_base.cc:1034] running on GCE node
I20260929 03:18:28.061650 31497 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:18:28.061896 31497 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:18:28.063041 31497 hybrid_clock.cc:648] HybridClock initialized: now 1790651908063031 us; error 32 us; skew 500 ppm
Sep 29 03:18:28 dist-test-slave-c9qh krb5kdc[31390](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651908, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:18:28.065591 31497 init.cc:377] Logged in from keytab as kudu/127.20.178.65@KRBTEST.COM (short username kudu)
I20260929 03:18:28.066071 31497 webserver.cc:460] Webserver started at http://127.20.178.65:43257/ using document root <none> and password file <none>
I20260929 03:18:28.066331 31497 fs_manager.cc:359] Metadata directory not provided
I20260929 03:18:28.066378 31497 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:18:28.066546 31497 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:18:28.067615 31497 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651852476208-21193-0/minicluster-data/ts-0/data/instance:
uuid: "0832e6f8c5684bc5a0b08ff65668b631"
format_stamp: "Formatted at 2026-09-29 03:18:28 on dist-test-slave-c9qh"
server_key: "c958d164ab990f3b925b35ea6d27982d"
server_key_iv: "dac682499609843b5d054cc5aa0d0284"
server_key_version: "encryptionkey@0"
I20260929 03:18:28.067994 31497 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651852476208-21193-0/minicluster-data/ts-0/wal/instance:
uuid: "0832e6f8c5684bc5a0b08ff65668b631"
format_stamp: "Formatted at 2026-09-29 03:18:28 on dist-test-slave-c9qh"
server_key: "c958d164ab990f3b925b35ea6d27982d"
server_key_iv: "dac682499609843b5d054cc5aa0d0284"
server_key_version: "encryptionkey@0"
I20260929 03:18:28.069386 31497 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 03:18:28.070119 31512 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:18:28.070315 31497 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 03:18:28.070425 31497 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651852476208-21193-0/minicluster-data/ts-0/data,/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651852476208-21193-0/minicluster-data/ts-0/wal
uuid: "0832e6f8c5684bc5a0b08ff65668b631"
format_stamp: "Formatted at 2026-09-29 03:18:28 on dist-test-slave-c9qh"
server_key: "c958d164ab990f3b925b35ea6d27982d"
server_key_iv: "dac682499609843b5d054cc5aa0d0284"
server_key_version: "encryptionkey@0"
I20260929 03:18:28.070519 31497 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651852476208-21193-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651852476208-21193-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651852476208-21193-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:18:28.082578 31497 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:18:28.084375 31497 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:18:28.084532 31497 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:18:28.084841 31497 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:18:28.085148 31497 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:18:28.085204 31497 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:18:28.085246 31497 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:18:28.085270 31497 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:18:28.090832 31497 rpc_server.cc:307] RPC server started. Bound to: 127.20.178.65:43035
I20260929 03:18:28.090910 31625 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.178.65:43035 every 8 connection(s)
I20260929 03:18:28.091223 31497 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651852476208-21193-0/minicluster-data/ts-0/data/info.pb
I20260929 03:18:28.095825 21193 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuoaHbX/build/release/bin/kudu as pid 31497
I20260929 03:18:28.095929 21193 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651852476208-21193-0/minicluster-data/ts-0/wal/instance
I20260929 03:18:28.096154 21193 external_mini_cluster.cc:1428] Setting key e372fb4e81b32511b8711fc0470db207
Sep 29 03:18:28 dist-test-slave-c9qh krb5kdc[31390](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651908, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.65@KRBTEST.COM for kudu/127.20.178.126@KRBTEST.COM
I20260929 03:18:28.100962 31475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:28.091931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.65:40957 (local address 127.20.178.126:41707)
0929 03:18:28.091994 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0929 03:18:28.091996 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:18:28.092184 (+ 188us) server_negotiation.cc:407] Connection header received
0929 03:18:28.092747 (+ 563us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:18:28.092749 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:18:28.092761 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:18:28.092808 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:18:28.093711 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:28.094213 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:28.095264 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:28.095481 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:28.095598 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:28.097981 (+ 2383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:18:28.097982 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:18:28.097983 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:18:28.097991 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:18:28.099398 (+ 1407us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:18:28.099886 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:18:28.099887 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:18:28.099887 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:18:28.099943 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:18:28.100233 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:18:28.100234 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:18:28.100234 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:18:28.100334 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:18:28.100386 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0929 03:18:28.100773 (+ 387us) server_negotiation.cc:299] Negotiation successful
0929 03:18:28.100854 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260929 03:18:28.101209 31628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:28.091922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:41707 (local address 127.20.178.65:40957)
0929 03:18:28.092069 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:28.092103 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0929 03:18:28.092615 (+ 512us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:18:28.092874 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:18:28.092878 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:18:28.093426 (+ 548us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:18:28.093610 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:28.093615 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:28.094318 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:28.094319 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:28.095165 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:28.095168 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:28.095606 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:28.095607 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:28.095754 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:28.096314 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:18:28.096324 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:18:28.097887 (+ 1563us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:18:28.099499 (+ 1612us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:18:28.099506 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:18:28.099509 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:18:28.099806 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:18:28.100019 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:18:28.100019 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:18:28.100020 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:18:28.100166 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:18:28.100398 (+ 232us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:18:28.100404 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:18:28.100570 (+ 166us) client_negotiation.cc:769] Sending connection context
0929 03:18:28.100743 (+ 173us) client_negotiation.cc:240] Negotiation successful
0929 03:18:28.100924 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":70,"threads_started":1}
I20260929 03:18:28.101775 31626 heartbeater.cc:347] Connected to a master server at 127.20.178.126:41707
I20260929 03:18:28.101883 31626 heartbeater.cc:464] Registering TS with master...
I20260929 03:18:28.102083 31626 heartbeater.cc:511] Master 127.20.178.126:41707 requested a full tablet report, sending...
I20260929 03:18:28.102614 31435 ts_manager.cc:194] Registered new tserver with Master: 0832e6f8c5684bc5a0b08ff65668b631 (127.20.178.65:43035)
I20260929 03:18:28.103267 31435 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.178.65@KRBTEST.COM'} at 127.20.178.65:40957
WARNING: no policy specified for kudu/127.20.178.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.178.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.178.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.66.keytab.
Entry for principal kudu/127.20.178.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.66.keytab.
I20260929 03:18:28.121721 21193 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuoaHbX/build/release/bin/kudu
/tmp/dist-test-taskuoaHbX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651852476208-21193-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651852476208-21193-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651852476208-21193-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651852476208-21193-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.178.66:0
--local_ip_for_outbound_sockets=127.20.178.66
--webserver_interface=127.20.178.66
--webserver_port=0
--tserver_master_addrs=127.20.178.126:41707
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.66.keytab
--principal=kudu/127.20.178.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.178.84:46455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651852476208-21193-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651852476208-21193-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651852476208-21193-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:18:28.201767 31631 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:18:28.201961 31631 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:18:28.201983 31631 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:18:28.203037 31631 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:18:28.203070 31631 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:18:28.203095 31631 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.178.66
I20260929 03:18:28.204422 31631 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.178.84:46455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651852476208-21193-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651852476208-21193-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.66.keytab
--principal=kudu/127.20.178.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.178.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651852476208-21193-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.178.66
--webserver_port=0
--tserver_master_addrs=127.20.178.126:41707
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31631
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.178.66
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651852476208-21193-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:18:28.204591 31631 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:18:28.204831 31631 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:18:28.209533 31639 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:18:28.209602 31636 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:18:28.209612 31637 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:18:28.209924 31631 server_base.cc:1034] running on GCE node
I20260929 03:18:28.210091 31631 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:18:28.210283 31631 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:18:28.211416 31631 hybrid_clock.cc:648] HybridClock initialized: now 1790651908211407 us; error 28 us; skew 500 ppm
Sep 29 03:18:28 dist-test-slave-c9qh krb5kdc[31390](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651908, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:18:28.213829 31631 init.cc:377] Logged in from keytab as kudu/127.20.178.66@KRBTEST.COM (short username kudu)
I20260929 03:18:28.214233 31631 webserver.cc:460] Webserver started at http://127.20.178.66:44747/ using document root <none> and password file <none>
I20260929 03:18:28.214455 31631 fs_manager.cc:359] Metadata directory not provided
I20260929 03:18:28.214520 31631 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:18:28.214663 31631 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:18:28.215675 31631 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651852476208-21193-0/minicluster-data/ts-1/data/instance:
uuid: "17a7225911ee45cab7c491dc1cbae7b8"
format_stamp: "Formatted at 2026-09-29 03:18:28 on dist-test-slave-c9qh"
server_key: "e909defeebf8383d36e42774ce623dd6"
server_key_iv: "a44e68da0daa5eef8a4c73390a8e7a48"
server_key_version: "encryptionkey@0"
I20260929 03:18:28.216024 31631 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651852476208-21193-0/minicluster-data/ts-1/wal/instance:
uuid: "17a7225911ee45cab7c491dc1cbae7b8"
format_stamp: "Formatted at 2026-09-29 03:18:28 on dist-test-slave-c9qh"
server_key: "e909defeebf8383d36e42774ce623dd6"
server_key_iv: "a44e68da0daa5eef8a4c73390a8e7a48"
server_key_version: "encryptionkey@0"
I20260929 03:18:28.217384 31631 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 03:18:28.218073 31646 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:18:28.218245 31631 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260929 03:18:28.218339 31631 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651852476208-21193-0/minicluster-data/ts-1/data,/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651852476208-21193-0/minicluster-data/ts-1/wal
uuid: "17a7225911ee45cab7c491dc1cbae7b8"
format_stamp: "Formatted at 2026-09-29 03:18:28 on dist-test-slave-c9qh"
server_key: "e909defeebf8383d36e42774ce623dd6"
server_key_iv: "a44e68da0daa5eef8a4c73390a8e7a48"
server_key_version: "encryptionkey@0"
I20260929 03:18:28.218413 31631 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651852476208-21193-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651852476208-21193-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651852476208-21193-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:18:28.230508 31631 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:18:28.232847 31631 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:18:28.232998 31631 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:18:28.233212 31631 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:18:28.233510 31631 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:18:28.233558 31631 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:18:28.233598 31631 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:18:28.233626 31631 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:18:28.239161 31631 rpc_server.cc:307] RPC server started. Bound to: 127.20.178.66:43733
I20260929 03:18:28.239205 31759 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.178.66:43733 every 8 connection(s)
I20260929 03:18:28.239494 31631 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651852476208-21193-0/minicluster-data/ts-1/data/info.pb
Sep 29 03:18:28 dist-test-slave-c9qh krb5kdc[31390](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651908, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.66@KRBTEST.COM for kudu/127.20.178.126@KRBTEST.COM
I20260929 03:18:28.246031 21193 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuoaHbX/build/release/bin/kudu as pid 31631
I20260929 03:18:28.246102 21193 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651852476208-21193-0/minicluster-data/ts-1/wal/instance
I20260929 03:18:28.246315 21193 external_mini_cluster.cc:1428] Setting key c323f4d4c1d212171cce0d5ee44817fc
I20260929 03:18:28.249143 31475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:28.240204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.66:41155 (local address 127.20.178.126:41707)
0929 03:18:28.240280 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0929 03:18:28.240282 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:18:28.240452 (+ 170us) server_negotiation.cc:407] Connection header received
0929 03:18:28.240918 (+ 466us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:18:28.240919 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:18:28.240928 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:18:28.240992 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:18:28.241842 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:28.242340 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:28.243245 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:28.243422 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:28.243471 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:28.245519 (+ 2048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:18:28.245520 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:18:28.245520 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:18:28.245526 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:18:28.247786 (+ 2260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:18:28.248155 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:18:28.248156 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:18:28.248157 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:18:28.248219 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:18:28.248464 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:18:28.248465 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:18:28.248465 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:18:28.248619 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:18:28.248713 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0929 03:18:28.249019 (+ 306us) server_negotiation.cc:299] Negotiation successful
0929 03:18:28.249082 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 03:18:28.249444 31762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:28.240165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:41707 (local address 127.20.178.66:41155)
0929 03:18:28.240329 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:28.240369 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 03:18:28.240841 (+ 472us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:18:28.241044 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:18:28.241047 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:18:28.241581 (+ 534us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:18:28.241767 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:28.241771 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:28.242418 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:28.242419 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:28.243173 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:28.243175 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:28.243490 (+ 315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:28.243491 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:28.243583 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:28.244140 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:18:28.244148 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:18:28.245431 (+ 1283us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:18:28.247864 (+ 2433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:18:28.247870 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:18:28.247872 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:18:28.248083 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:18:28.248286 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:18:28.248286 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:18:28.248286 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:18:28.248380 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:18:28.248704 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:18:28.248710 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:18:28.248878 (+ 168us) client_negotiation.cc:769] Sending connection context
0929 03:18:28.249001 (+ 123us) client_negotiation.cc:240] Negotiation successful
0929 03:18:28.249172 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":99,"threads_started":1}
I20260929 03:18:28.249912 31760 heartbeater.cc:347] Connected to a master server at 127.20.178.126:41707
I20260929 03:18:28.250020 31760 heartbeater.cc:464] Registering TS with master...
I20260929 03:18:28.250200 31760 heartbeater.cc:511] Master 127.20.178.126:41707 requested a full tablet report, sending...
I20260929 03:18:28.250607 31435 ts_manager.cc:194] Registered new tserver with Master: 17a7225911ee45cab7c491dc1cbae7b8 (127.20.178.66:43733)
I20260929 03:18:28.251084 31435 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.178.66@KRBTEST.COM'} at 127.20.178.66:41155
WARNING: no policy specified for kudu/127.20.178.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.178.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.178.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.67.keytab.
Entry for principal kudu/127.20.178.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.67.keytab.
I20260929 03:18:28.271497 21193 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuoaHbX/build/release/bin/kudu
/tmp/dist-test-taskuoaHbX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651852476208-21193-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651852476208-21193-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651852476208-21193-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651852476208-21193-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.178.67:0
--local_ip_for_outbound_sockets=127.20.178.67
--webserver_interface=127.20.178.67
--webserver_port=0
--tserver_master_addrs=127.20.178.126:41707
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.67.keytab
--principal=kudu/127.20.178.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.178.84:46455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651852476208-21193-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651852476208-21193-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651852476208-21193-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:18:28.352842 31765 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:18:28.353045 31765 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:18:28.353068 31765 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:18:28.354245 31765 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:18:28.354286 31765 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:18:28.354313 31765 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.178.67
I20260929 03:18:28.355729 31765 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.178.84:46455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651852476208-21193-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651852476208-21193-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.67.keytab
--principal=kudu/127.20.178.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.178.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651852476208-21193-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.178.67
--webserver_port=0
--tserver_master_addrs=127.20.178.126:41707
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31765
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.178.67
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651852476208-21193-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:18:28.355962 31765 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:18:28.356196 31765 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:18:28.360957 31773 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:18:28.360962 31771 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:18:28.360962 31770 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:18:28.361280 31765 server_base.cc:1034] running on GCE node
I20260929 03:18:28.361490 31765 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:18:28.361726 31765 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:18:28.362888 31765 hybrid_clock.cc:648] HybridClock initialized: now 1790651908362871 us; error 36 us; skew 500 ppm
Sep 29 03:18:28 dist-test-slave-c9qh krb5kdc[31390](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651908, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:18:28.365233 31765 init.cc:377] Logged in from keytab as kudu/127.20.178.67@KRBTEST.COM (short username kudu)
I20260929 03:18:28.365646 31765 webserver.cc:460] Webserver started at http://127.20.178.67:35591/ using document root <none> and password file <none>
I20260929 03:18:28.365826 31765 fs_manager.cc:359] Metadata directory not provided
I20260929 03:18:28.365865 31765 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:18:28.365948 31765 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:18:28.366912 31765 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651852476208-21193-0/minicluster-data/ts-2/data/instance:
uuid: "5acd4aea3b3a41a4810d08531f175532"
format_stamp: "Formatted at 2026-09-29 03:18:28 on dist-test-slave-c9qh"
server_key: "7ae54c7383badb07edb030a0b0ff0464"
server_key_iv: "0bb0301ec84b9dfd388812b120677f3e"
server_key_version: "encryptionkey@0"
I20260929 03:18:28.367197 31765 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651852476208-21193-0/minicluster-data/ts-2/wal/instance:
uuid: "5acd4aea3b3a41a4810d08531f175532"
format_stamp: "Formatted at 2026-09-29 03:18:28 on dist-test-slave-c9qh"
server_key: "7ae54c7383badb07edb030a0b0ff0464"
server_key_iv: "0bb0301ec84b9dfd388812b120677f3e"
server_key_version: "encryptionkey@0"
I20260929 03:18:28.368422 31765 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 03:18:28.369140 31780 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:18:28.369306 31765 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260929 03:18:28.369390 31765 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651852476208-21193-0/minicluster-data/ts-2/data,/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651852476208-21193-0/minicluster-data/ts-2/wal
uuid: "5acd4aea3b3a41a4810d08531f175532"
format_stamp: "Formatted at 2026-09-29 03:18:28 on dist-test-slave-c9qh"
server_key: "7ae54c7383badb07edb030a0b0ff0464"
server_key_iv: "0bb0301ec84b9dfd388812b120677f3e"
server_key_version: "encryptionkey@0"
I20260929 03:18:28.369462 31765 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651852476208-21193-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651852476208-21193-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651852476208-21193-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:18:28.382471 31765 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:18:28.384765 31765 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:18:28.384917 31765 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:18:28.385139 31765 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:18:28.385450 31765 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:18:28.385483 31765 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:18:28.385535 31765 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:18:28.385561 31765 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:18:28.391366 31765 rpc_server.cc:307] RPC server started. Bound to: 127.20.178.67:45619
I20260929 03:18:28.391436 31893 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.178.67:45619 every 8 connection(s)
I20260929 03:18:28.391768 31765 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651852476208-21193-0/minicluster-data/ts-2/data/info.pb
I20260929 03:18:28.396054 21193 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuoaHbX/build/release/bin/kudu as pid 31765
I20260929 03:18:28.396173 21193 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651852476208-21193-0/minicluster-data/ts-2/wal/instance
I20260929 03:18:28.396364 21193 external_mini_cluster.cc:1428] Setting key 50cf6659a990f12dc79a1a8a9ad52e4e
Sep 29 03:18:28 dist-test-slave-c9qh krb5kdc[31390](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651908, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.67@KRBTEST.COM for kudu/127.20.178.126@KRBTEST.COM
I20260929 03:18:28.400748 31475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:28.392537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.67:55157 (local address 127.20.178.126:41707)
0929 03:18:28.392579 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0929 03:18:28.392582 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:18:28.392793 (+ 211us) server_negotiation.cc:407] Connection header received
0929 03:18:28.393305 (+ 512us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:18:28.393306 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:18:28.393326 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:18:28.393382 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:18:28.394303 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:28.394802 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:28.395784 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:28.396064 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:28.396140 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:28.398384 (+ 2244us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:18:28.398385 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:18:28.398385 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:18:28.398392 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:18:28.399584 (+ 1192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:18:28.399934 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:18:28.399935 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:18:28.399935 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:18:28.399981 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:18:28.400191 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:18:28.400192 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:18:28.400192 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:18:28.400284 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:18:28.400333 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0929 03:18:28.400553 (+ 220us) server_negotiation.cc:299] Negotiation successful
0929 03:18:28.400608 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260929 03:18:28.400893 31896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:28.392491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:41707 (local address 127.20.178.67:55157)
0929 03:18:28.392664 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:28.392709 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 03:18:28.393201 (+ 492us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:18:28.393446 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:18:28.393450 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:18:28.394007 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:18:28.394194 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:28.394198 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:28.394894 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:28.394895 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:28.395678 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:28.395680 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:28.396178 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:28.396180 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:28.396332 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:28.396954 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:18:28.396962 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:18:28.398297 (+ 1335us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:18:28.399657 (+ 1360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:18:28.399663 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:18:28.399664 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:18:28.399869 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:18:28.400037 (+ 168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:18:28.400038 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:18:28.400038 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:18:28.400133 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:18:28.400338 (+ 205us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:18:28.400342 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:18:28.400449 (+ 107us) client_negotiation.cc:769] Sending connection context
0929 03:18:28.400528 (+ 79us) client_negotiation.cc:240] Negotiation successful
0929 03:18:28.400700 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":69,"threads_started":1}
I20260929 03:18:28.401306 31894 heartbeater.cc:347] Connected to a master server at 127.20.178.126:41707
I20260929 03:18:28.401382 31894 heartbeater.cc:464] Registering TS with master...
I20260929 03:18:28.401553 31894 heartbeater.cc:511] Master 127.20.178.126:41707 requested a full tablet report, sending...
I20260929 03:18:28.401902 31435 ts_manager.cc:194] Registered new tserver with Master: 5acd4aea3b3a41a4810d08531f175532 (127.20.178.67:45619)
I20260929 03:18:28.402281 31435 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.178.67@KRBTEST.COM'} at 127.20.178.67:55157
I20260929 03:18:28.407919 21193 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 03:18:28.415547 31475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:28.409313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42944 (local address 127.20.178.126:41707)
0929 03:18:28.409381 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0929 03:18:28.409383 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:18:28.409472 (+ 89us) server_negotiation.cc:407] Connection header received
0929 03:18:28.409559 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:18:28.409561 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:18:28.409570 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:18:28.409640 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:18:28.410186 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:28.410701 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:28.411440 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:28.411678 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:28.411765 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:28.413054 (+ 1289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:18:28.413055 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:18:28.413056 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:18:28.413063 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:18:28.414411 (+ 1348us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:18:28.414756 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:18:28.414757 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:18:28.414757 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:18:28.414810 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:18:28.415010 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:18:28.415011 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:18:28.415012 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:18:28.415155 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:18:28.415262 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0929 03:18:28.415374 (+ 112us) server_negotiation.cc:299] Negotiation successful
0929 03:18:28.415451 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260929 03:18:28.417117 31435 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42944:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 03:18:28.418566 31435 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 03:18:28.424921 31906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:28.420782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60522 (local address 127.20.178.66:43733)
0929 03:18:28.420974 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0929 03:18:28.420980 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:18:28.420993 (+ 13us) server_negotiation.cc:407] Connection header received
0929 03:18:28.421115 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:18:28.421118 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:18:28.421136 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:18:28.421214 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:18:28.421372 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:28.422594 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:28.423848 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:28.424499 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:28.424565 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:28.424622 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0929 03:18:28.424806 (+ 184us) server_negotiation.cc:299] Negotiation successful
0929 03:18:28.424846 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":81,"threads_started":1}
I20260929 03:18:28.424949 31904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:28.420576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.66:43733 (local address 127.0.0.1:60522)
0929 03:18:28.420877 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:28.420923 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 03:18:28.420988 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:18:28.421215 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:18:28.421218 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:18:28.421220 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:18:28.421284 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:28.421286 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:28.422696 (+ 1410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:28.422697 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:28.423776 (+ 1079us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:28.423777 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:28.424578 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:28.424579 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:28.424689 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:28.424705 (+ 16us) client_negotiation.cc:769] Sending connection context
0929 03:18:28.424783 (+ 78us) client_negotiation.cc:240] Negotiation successful
0929 03:18:28.424851 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"mutex_wait_us":4,"thread_start_us":87,"threads_started":1}
I20260929 03:18:28.424949 31903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:28.420577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.67:45619 (local address 127.0.0.1:46510)
0929 03:18:28.421032 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:28.421043 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 03:18:28.421102 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:18:28.421437 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:18:28.421438 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:18:28.421440 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:18:28.421487 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:28.421488 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:28.422758 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:28.422759 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:28.423678 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:28.423680 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:28.424580 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:28.424580 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:28.424689 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:28.424705 (+ 16us) client_negotiation.cc:769] Sending connection context
0929 03:18:28.424783 (+ 78us) client_negotiation.cc:240] Negotiation successful
0929 03:18:28.424852 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":442,"mutex_wait_us":27,"thread_start_us":77,"threads_started":1}
I20260929 03:18:28.425452 31905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:28.420778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46510 (local address 127.20.178.67:45619)
0929 03:18:28.420985 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0929 03:18:28.420989 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:18:28.421160 (+ 171us) server_negotiation.cc:407] Connection header received
0929 03:18:28.421214 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:18:28.421216 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:18:28.421231 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:18:28.421367 (+ 136us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:18:28.421591 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:28.422680 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:28.423761 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:28.424487 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:28.424565 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:28.424622 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0929 03:18:28.425337 (+ 715us) server_negotiation.cc:299] Negotiation successful
0929 03:18:28.425380 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":85,"threads_started":1}
I20260929 03:18:28.425841 31694 tablet_service.cc:1467] Processing CreateTablet for tablet 105f3cff6c854e36ac35c3b82eb7b388 (DEFAULT_TABLE table=test-table [id=05ed7055c156480d9df661cd62689db1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:18:28.426074 31828 tablet_service.cc:1467] Processing CreateTablet for tablet 105f3cff6c854e36ac35c3b82eb7b388 (DEFAULT_TABLE table=test-table [id=05ed7055c156480d9df661cd62689db1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:18:28.426222 31694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 105f3cff6c854e36ac35c3b82eb7b388. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:18:28.426405 31828 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 105f3cff6c854e36ac35c3b82eb7b388. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:18:28.428743 31908 tablet_bootstrap.cc:492] T 105f3cff6c854e36ac35c3b82eb7b388 P 5acd4aea3b3a41a4810d08531f175532: Bootstrap starting.
I20260929 03:18:28.429214 31909 tablet_bootstrap.cc:492] T 105f3cff6c854e36ac35c3b82eb7b388 P 17a7225911ee45cab7c491dc1cbae7b8: Bootstrap starting.
I20260929 03:18:28.429456 31907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:28.420838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60708 (local address 127.20.178.65:43035)
0929 03:18:28.421062 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0929 03:18:28.421067 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:18:28.421096 (+ 29us) server_negotiation.cc:407] Connection header received
0929 03:18:28.421168 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:18:28.421170 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:18:28.421184 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:18:28.421262 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:18:28.421466 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:28.422361 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:28.424694 (+ 2333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:28.429016 (+ 4322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:28.429073 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:28.429110 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0929 03:18:28.429300 (+ 190us) server_negotiation.cc:299] Negotiation successful
0929 03:18:28.429349 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":74,"threads_started":1}
I20260929 03:18:28.429513 31908 tablet_bootstrap.cc:654] T 105f3cff6c854e36ac35c3b82eb7b388 P 5acd4aea3b3a41a4810d08531f175532: Neither blocks nor log segments found. Creating new log.
I20260929 03:18:28.429678 31902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:28.420584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.65:43035 (local address 127.0.0.1:60708)
0929 03:18:28.421034 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:28.421043 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 03:18:28.421103 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:18:28.421338 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:18:28.421339 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:18:28.421341 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:18:28.421387 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:28.421388 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:28.422459 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:28.422460 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:28.423769 (+ 1309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:28.423772 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:28.429098 (+ 5326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:28.429099 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:28.429181 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:28.429188 (+ 7us) client_negotiation.cc:769] Sending connection context
0929 03:18:28.429550 (+ 362us) client_negotiation.cc:240] Negotiation successful
0929 03:18:28.429609 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":442,"mutex_wait_us":23,"thread_start_us":108,"threads_started":1}
I20260929 03:18:28.429945 31909 tablet_bootstrap.cc:654] T 105f3cff6c854e36ac35c3b82eb7b388 P 17a7225911ee45cab7c491dc1cbae7b8: Neither blocks nor log segments found. Creating new log.
I20260929 03:18:28.430315 31908 log.cc:824] T 105f3cff6c854e36ac35c3b82eb7b388 P 5acd4aea3b3a41a4810d08531f175532: Log is configured to *not* fsync() on all Append() calls
I20260929 03:18:28.430541 31909 log.cc:824] T 105f3cff6c854e36ac35c3b82eb7b388 P 17a7225911ee45cab7c491dc1cbae7b8: Log is configured to *not* fsync() on all Append() calls
I20260929 03:18:28.430518 31560 tablet_service.cc:1467] Processing CreateTablet for tablet 105f3cff6c854e36ac35c3b82eb7b388 (DEFAULT_TABLE table=test-table [id=05ed7055c156480d9df661cd62689db1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:18:28.430886 31560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 105f3cff6c854e36ac35c3b82eb7b388. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:18:28.431147 31908 tablet_bootstrap.cc:492] T 105f3cff6c854e36ac35c3b82eb7b388 P 5acd4aea3b3a41a4810d08531f175532: No bootstrap required, opened a new log
I20260929 03:18:28.431232 31908 ts_tablet_manager.cc:1397] T 105f3cff6c854e36ac35c3b82eb7b388 P 5acd4aea3b3a41a4810d08531f175532: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260929 03:18:28.431332 31909 tablet_bootstrap.cc:492] T 105f3cff6c854e36ac35c3b82eb7b388 P 17a7225911ee45cab7c491dc1cbae7b8: No bootstrap required, opened a new log
I20260929 03:18:28.431543 31909 ts_tablet_manager.cc:1397] T 105f3cff6c854e36ac35c3b82eb7b388 P 17a7225911ee45cab7c491dc1cbae7b8: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 03:18:28.432906 31908 raft_consensus.cc:348] T 105f3cff6c854e36ac35c3b82eb7b388 P 5acd4aea3b3a41a4810d08531f175532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17a7225911ee45cab7c491dc1cbae7b8" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 43733 } } peers { permanent_uuid: "5acd4aea3b3a41a4810d08531f175532" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 45619 } } peers { permanent_uuid: "0832e6f8c5684bc5a0b08ff65668b631" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 43035 } }
I20260929 03:18:28.433059 31908 raft_consensus.cc:374] T 105f3cff6c854e36ac35c3b82eb7b388 P 5acd4aea3b3a41a4810d08531f175532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:18:28.433120 31908 raft_consensus.cc:729] T 105f3cff6c854e36ac35c3b82eb7b388 P 5acd4aea3b3a41a4810d08531f175532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5acd4aea3b3a41a4810d08531f175532, State: Initialized, Role: FOLLOWER
I20260929 03:18:28.433248 31908 consensus_queue.cc:260] T 105f3cff6c854e36ac35c3b82eb7b388 P 5acd4aea3b3a41a4810d08531f175532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17a7225911ee45cab7c491dc1cbae7b8" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 43733 } } peers { permanent_uuid: "5acd4aea3b3a41a4810d08531f175532" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 45619 } } peers { permanent_uuid: "0832e6f8c5684bc5a0b08ff65668b631" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 43035 } }
I20260929 03:18:28.433523 31908 ts_tablet_manager.cc:1428] T 105f3cff6c854e36ac35c3b82eb7b388 P 5acd4aea3b3a41a4810d08531f175532: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260929 03:18:28.433646 31894 heartbeater.cc:503] Master 127.20.178.126:41707 was elected leader, sending a full tablet report...
I20260929 03:18:28.433635 31909 raft_consensus.cc:348] T 105f3cff6c854e36ac35c3b82eb7b388 P 17a7225911ee45cab7c491dc1cbae7b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17a7225911ee45cab7c491dc1cbae7b8" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 43733 } } peers { permanent_uuid: "5acd4aea3b3a41a4810d08531f175532" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 45619 } } peers { permanent_uuid: "0832e6f8c5684bc5a0b08ff65668b631" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 43035 } }
I20260929 03:18:28.433812 31909 raft_consensus.cc:374] T 105f3cff6c854e36ac35c3b82eb7b388 P 17a7225911ee45cab7c491dc1cbae7b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:18:28.433863 31909 raft_consensus.cc:729] T 105f3cff6c854e36ac35c3b82eb7b388 P 17a7225911ee45cab7c491dc1cbae7b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17a7225911ee45cab7c491dc1cbae7b8, State: Initialized, Role: FOLLOWER
I20260929 03:18:28.434280 31913 tablet_bootstrap.cc:492] T 105f3cff6c854e36ac35c3b82eb7b388 P 0832e6f8c5684bc5a0b08ff65668b631: Bootstrap starting.
I20260929 03:18:28.434441 31909 consensus_queue.cc:260] T 105f3cff6c854e36ac35c3b82eb7b388 P 17a7225911ee45cab7c491dc1cbae7b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17a7225911ee45cab7c491dc1cbae7b8" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 43733 } } peers { permanent_uuid: "5acd4aea3b3a41a4810d08531f175532" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 45619 } } peers { permanent_uuid: "0832e6f8c5684bc5a0b08ff65668b631" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 43035 } }
I20260929 03:18:28.434764 31909 ts_tablet_manager.cc:1428] T 105f3cff6c854e36ac35c3b82eb7b388 P 17a7225911ee45cab7c491dc1cbae7b8: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260929 03:18:28.434795 31760 heartbeater.cc:503] Master 127.20.178.126:41707 was elected leader, sending a full tablet report...
I20260929 03:18:28.434942 31913 tablet_bootstrap.cc:654] T 105f3cff6c854e36ac35c3b82eb7b388 P 0832e6f8c5684bc5a0b08ff65668b631: Neither blocks nor log segments found. Creating new log.
I20260929 03:18:28.435544 31913 log.cc:824] T 105f3cff6c854e36ac35c3b82eb7b388 P 0832e6f8c5684bc5a0b08ff65668b631: Log is configured to *not* fsync() on all Append() calls
I20260929 03:18:28.436261 31913 tablet_bootstrap.cc:492] T 105f3cff6c854e36ac35c3b82eb7b388 P 0832e6f8c5684bc5a0b08ff65668b631: No bootstrap required, opened a new log
I20260929 03:18:28.436343 31913 ts_tablet_manager.cc:1397] T 105f3cff6c854e36ac35c3b82eb7b388 P 0832e6f8c5684bc5a0b08ff65668b631: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 03:18:28.437669 31913 raft_consensus.cc:348] T 105f3cff6c854e36ac35c3b82eb7b388 P 0832e6f8c5684bc5a0b08ff65668b631 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17a7225911ee45cab7c491dc1cbae7b8" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 43733 } } peers { permanent_uuid: "5acd4aea3b3a41a4810d08531f175532" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 45619 } } peers { permanent_uuid: "0832e6f8c5684bc5a0b08ff65668b631" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 43035 } }
I20260929 03:18:28.437829 31913 raft_consensus.cc:374] T 105f3cff6c854e36ac35c3b82eb7b388 P 0832e6f8c5684bc5a0b08ff65668b631 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:18:28.437866 31913 raft_consensus.cc:729] T 105f3cff6c854e36ac35c3b82eb7b388 P 0832e6f8c5684bc5a0b08ff65668b631 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0832e6f8c5684bc5a0b08ff65668b631, State: Initialized, Role: FOLLOWER
I20260929 03:18:28.437978 31913 consensus_queue.cc:260] T 105f3cff6c854e36ac35c3b82eb7b388 P 0832e6f8c5684bc5a0b08ff65668b631 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17a7225911ee45cab7c491dc1cbae7b8" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 43733 } } peers { permanent_uuid: "5acd4aea3b3a41a4810d08531f175532" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 45619 } } peers { permanent_uuid: "0832e6f8c5684bc5a0b08ff65668b631" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 43035 } }
I20260929 03:18:28.438234 31913 ts_tablet_manager.cc:1428] T 105f3cff6c854e36ac35c3b82eb7b388 P 0832e6f8c5684bc5a0b08ff65668b631: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260929 03:18:28.438257 31626 heartbeater.cc:503] Master 127.20.178.126:41707 was elected leader, sending a full tablet report...
I20260929 03:18:28.445493 31912 raft_consensus.cc:482] T 105f3cff6c854e36ac35c3b82eb7b388 P 5acd4aea3b3a41a4810d08531f175532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:18:28.445574 31912 raft_consensus.cc:504] T 105f3cff6c854e36ac35c3b82eb7b388 P 5acd4aea3b3a41a4810d08531f175532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17a7225911ee45cab7c491dc1cbae7b8" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 43733 } } peers { permanent_uuid: "5acd4aea3b3a41a4810d08531f175532" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 45619 } } peers { permanent_uuid: "0832e6f8c5684bc5a0b08ff65668b631" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 43035 } }
I20260929 03:18:28.445958 31912 leader_election.cc:288] T 105f3cff6c854e36ac35c3b82eb7b388 P 5acd4aea3b3a41a4810d08531f175532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 17a7225911ee45cab7c491dc1cbae7b8 (127.20.178.66:43733), 0832e6f8c5684bc5a0b08ff65668b631 (127.20.178.65:43035)
I20260929 03:18:28.449713 31906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:28.446198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.67:41987 (local address 127.20.178.66:43733)
0929 03:18:28.446259 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0929 03:18:28.446261 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:18:28.446271 (+ 10us) server_negotiation.cc:407] Connection header received
0929 03:18:28.446321 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:18:28.446322 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:18:28.446332 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:18:28.446388 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:18:28.446574 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:28.447355 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:28.448785 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:28.449369 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:28.449447 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:28.449456 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0929 03:18:28.449608 (+ 152us) server_negotiation.cc:299] Negotiation successful
0929 03:18:28.449645 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260929 03:18:28.449713 31907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:28.446198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.67:37389 (local address 127.20.178.65:43035)
0929 03:18:28.446256 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0929 03:18:28.446258 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:18:28.446562 (+ 304us) server_negotiation.cc:407] Connection header received
0929 03:18:28.446587 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:18:28.446589 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:18:28.446597 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:18:28.446649 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:18:28.446767 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:28.447431 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:28.448785 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:28.449369 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:28.449447 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:28.449456 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0929 03:18:28.449608 (+ 152us) server_negotiation.cc:299] Negotiation successful
0929 03:18:28.449645 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260929 03:18:28.449751 31917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:28.446134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.65:43035 (local address 127.20.178.67:37389)
0929 03:18:28.446473 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:28.446483 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 03:18:28.446545 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:18:28.446651 (+ 106us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:18:28.446651 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:18:28.446652 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:18:28.446703 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:28.446704 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:28.447494 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:28.447495 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:28.448698 (+ 1203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:28.448701 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:28.449452 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:28.449453 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:28.449528 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:28.449533 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 03:18:28.449589 (+ 56us) client_negotiation.cc:240] Negotiation successful
0929 03:18:28.449639 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"mutex_wait_us":199,"thread_start_us":86,"threads_started":1}
I20260929 03:18:28.449759 31896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:28.446134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.66:43733 (local address 127.20.178.67:41987)
0929 03:18:28.446175 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:28.446187 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 03:18:28.446256 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:18:28.446401 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:18:28.446402 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:18:28.446405 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:18:28.446494 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:28.446497 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:28.447427 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:28.447428 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:28.448698 (+ 1270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:28.448701 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:28.449452 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:28.449452 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:28.449531 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:28.449533 (+ 2us) client_negotiation.cc:769] Sending connection context
0929 03:18:28.449589 (+ 56us) client_negotiation.cc:240] Negotiation successful
0929 03:18:28.449639 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23,"mutex_wait_us":36}
I20260929 03:18:28.450245 31714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "105f3cff6c854e36ac35c3b82eb7b388" candidate_uuid: "5acd4aea3b3a41a4810d08531f175532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17a7225911ee45cab7c491dc1cbae7b8" is_pre_election: true
I20260929 03:18:28.450253 31580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "105f3cff6c854e36ac35c3b82eb7b388" candidate_uuid: "5acd4aea3b3a41a4810d08531f175532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0832e6f8c5684bc5a0b08ff65668b631" is_pre_election: true
I20260929 03:18:28.450402 31580 raft_consensus.cc:2454] T 105f3cff6c854e36ac35c3b82eb7b388 P 0832e6f8c5684bc5a0b08ff65668b631 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5acd4aea3b3a41a4810d08531f175532 in term 0.
I20260929 03:18:28.450402 31714 raft_consensus.cc:2454] T 105f3cff6c854e36ac35c3b82eb7b388 P 17a7225911ee45cab7c491dc1cbae7b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5acd4aea3b3a41a4810d08531f175532 in term 0.
I20260929 03:18:28.450616 31784 leader_election.cc:302] T 105f3cff6c854e36ac35c3b82eb7b388 P 5acd4aea3b3a41a4810d08531f175532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0832e6f8c5684bc5a0b08ff65668b631, 5acd4aea3b3a41a4810d08531f175532; no voters:
I20260929 03:18:28.450781 31912 raft_consensus.cc:2781] T 105f3cff6c854e36ac35c3b82eb7b388 P 5acd4aea3b3a41a4810d08531f175532 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 03:18:28.450829 31912 raft_consensus.cc:482] T 105f3cff6c854e36ac35c3b82eb7b388 P 5acd4aea3b3a41a4810d08531f175532 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 03:18:28.450853 31912 raft_consensus.cc:3037] T 105f3cff6c854e36ac35c3b82eb7b388 P 5acd4aea3b3a41a4810d08531f175532 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:18:28.451526 31912 raft_consensus.cc:504] T 105f3cff6c854e36ac35c3b82eb7b388 P 5acd4aea3b3a41a4810d08531f175532 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17a7225911ee45cab7c491dc1cbae7b8" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 43733 } } peers { permanent_uuid: "5acd4aea3b3a41a4810d08531f175532" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 45619 } } peers { permanent_uuid: "0832e6f8c5684bc5a0b08ff65668b631" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 43035 } }
I20260929 03:18:28.451681 31912 leader_election.cc:288] T 105f3cff6c854e36ac35c3b82eb7b388 P 5acd4aea3b3a41a4810d08531f175532 [CANDIDATE]: Term 1 election: Requested vote from peers 17a7225911ee45cab7c491dc1cbae7b8 (127.20.178.66:43733), 0832e6f8c5684bc5a0b08ff65668b631 (127.20.178.65:43035)
I20260929 03:18:28.451890 31580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "105f3cff6c854e36ac35c3b82eb7b388" candidate_uuid: "5acd4aea3b3a41a4810d08531f175532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0832e6f8c5684bc5a0b08ff65668b631"
I20260929 03:18:28.451903 31714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "105f3cff6c854e36ac35c3b82eb7b388" candidate_uuid: "5acd4aea3b3a41a4810d08531f175532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17a7225911ee45cab7c491dc1cbae7b8"
I20260929 03:18:28.451972 31580 raft_consensus.cc:3037] T 105f3cff6c854e36ac35c3b82eb7b388 P 0832e6f8c5684bc5a0b08ff65668b631 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:18:28.451972 31714 raft_consensus.cc:3037] T 105f3cff6c854e36ac35c3b82eb7b388 P 17a7225911ee45cab7c491dc1cbae7b8 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:18:28.452906 31580 raft_consensus.cc:2454] T 105f3cff6c854e36ac35c3b82eb7b388 P 0832e6f8c5684bc5a0b08ff65668b631 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5acd4aea3b3a41a4810d08531f175532 in term 1.
I20260929 03:18:28.452906 31714 raft_consensus.cc:2454] T 105f3cff6c854e36ac35c3b82eb7b388 P 17a7225911ee45cab7c491dc1cbae7b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5acd4aea3b3a41a4810d08531f175532 in term 1.
I20260929 03:18:28.453068 31784 leader_election.cc:302] T 105f3cff6c854e36ac35c3b82eb7b388 P 5acd4aea3b3a41a4810d08531f175532 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0832e6f8c5684bc5a0b08ff65668b631, 5acd4aea3b3a41a4810d08531f175532; no voters:
I20260929 03:18:28.453192 31912 raft_consensus.cc:2781] T 105f3cff6c854e36ac35c3b82eb7b388 P 5acd4aea3b3a41a4810d08531f175532 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:18:28.453327 31912 raft_consensus.cc:686] T 105f3cff6c854e36ac35c3b82eb7b388 P 5acd4aea3b3a41a4810d08531f175532 [term 1 LEADER]: Becoming Leader. State: Replica: 5acd4aea3b3a41a4810d08531f175532, State: Running, Role: LEADER
I20260929 03:18:28.453420 31912 consensus_queue.cc:237] T 105f3cff6c854e36ac35c3b82eb7b388 P 5acd4aea3b3a41a4810d08531f175532 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17a7225911ee45cab7c491dc1cbae7b8" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 43733 } } peers { permanent_uuid: "5acd4aea3b3a41a4810d08531f175532" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 45619 } } peers { permanent_uuid: "0832e6f8c5684bc5a0b08ff65668b631" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 43035 } }
I20260929 03:18:28.454160 31434 catalog_manager.cc:5495] T 105f3cff6c854e36ac35c3b82eb7b388 P 5acd4aea3b3a41a4810d08531f175532 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5acd4aea3b3a41a4810d08531f175532 (127.20.178.67). New cstate: current_term: 1 leader_uuid: "5acd4aea3b3a41a4810d08531f175532" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17a7225911ee45cab7c491dc1cbae7b8" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 43733 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5acd4aea3b3a41a4810d08531f175532" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 45619 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0832e6f8c5684bc5a0b08ff65668b631" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 43035 } health_report { overall_health: UNKNOWN } } }
I20260929 03:18:28.461997 31905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:28.459853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46516 (local address 127.20.178.67:45619)
0929 03:18:28.459891 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0929 03:18:28.459893 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:18:28.459902 (+ 9us) server_negotiation.cc:407] Connection header received
0929 03:18:28.459960 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:18:28.459961 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:18:28.459975 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:18:28.460027 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:18:28.460147 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:28.460590 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:28.461352 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:28.461525 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:28.461573 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:28.461737 (+ 164us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:18:28.461826 (+ 89us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:18:28.461885 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0929 03:18:28.461916 (+ 31us) server_negotiation.cc:299] Negotiation successful
0929 03:18:28.461932 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260929 03:18:28.463629 31714 raft_consensus.cc:1261] T 105f3cff6c854e36ac35c3b82eb7b388 P 17a7225911ee45cab7c491dc1cbae7b8 [term 1 FOLLOWER]: Refusing update from remote peer 5acd4aea3b3a41a4810d08531f175532: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:18:28.463639 31580 raft_consensus.cc:1261] T 105f3cff6c854e36ac35c3b82eb7b388 P 0832e6f8c5684bc5a0b08ff65668b631 [term 1 FOLLOWER]: Refusing update from remote peer 5acd4aea3b3a41a4810d08531f175532: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:18:28.464001 31918 consensus_queue.cc:1035] T 105f3cff6c854e36ac35c3b82eb7b388 P 5acd4aea3b3a41a4810d08531f175532 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0832e6f8c5684bc5a0b08ff65668b631" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 43035 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:18:28.464138 31912 consensus_queue.cc:1035] T 105f3cff6c854e36ac35c3b82eb7b388 P 5acd4aea3b3a41a4810d08531f175532 [LEADER]: Connected to new peer: Peer: permanent_uuid: "17a7225911ee45cab7c491dc1cbae7b8" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 43733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:18:28.466717 31922 mvcc.cc:205] Tried to move back new op lower bound from 7334510217064497152 to 7334510217025925120. Current Snapshot: MvccSnapshot[applied={T|T < 7334510217064497152}]
I20260929 03:18:28.467177 31923 mvcc.cc:205] Tried to move back new op lower bound from 7334510217064497152 to 7334510217025925120. Current Snapshot: MvccSnapshot[applied={T|T < 7334510217064497152}]
I20260929 03:18:28.468657 31435 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42944:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:18:28.468753 31435 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42944:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:18:28.469506 31435 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 045d51f9ab9e4873a97eacfd153c3919: Sending DeleteTablet for 3 replicas of tablet 105f3cff6c854e36ac35c3b82eb7b388
I20260929 03:18:28.469822 31694 tablet_service.cc:1514] Processing DeleteTablet for tablet 105f3cff6c854e36ac35c3b82eb7b388 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:18:28 UTC) from {username='kudu', principal='kudu/127.20.178.126@KRBTEST.COM'} at 127.0.0.1:60522
I20260929 03:18:28.469852 31560 tablet_service.cc:1514] Processing DeleteTablet for tablet 105f3cff6c854e36ac35c3b82eb7b388 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:18:28 UTC) from {username='kudu', principal='kudu/127.20.178.126@KRBTEST.COM'} at 127.0.0.1:60708
I20260929 03:18:28.469871 31828 tablet_service.cc:1514] Processing DeleteTablet for tablet 105f3cff6c854e36ac35c3b82eb7b388 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:18:28 UTC) from {username='kudu', principal='kudu/127.20.178.126@KRBTEST.COM'} at 127.0.0.1:46510
I20260929 03:18:28.470278 31931 tablet_replica.cc:330] T 105f3cff6c854e36ac35c3b82eb7b388 P 0832e6f8c5684bc5a0b08ff65668b631: stopping tablet replica
I20260929 03:18:28.470283 21193 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuoaHbX/build/release/bin/kudu with pid 31497
I20260929 03:18:28.470098 31930 tablet_replica.cc:330] T 105f3cff6c854e36ac35c3b82eb7b388 P 17a7225911ee45cab7c491dc1cbae7b8: stopping tablet replica
I20260929 03:18:28.470908 31930 raft_consensus.cc:2229] T 105f3cff6c854e36ac35c3b82eb7b388 P 17a7225911ee45cab7c491dc1cbae7b8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:18:28.471014 31930 raft_consensus.cc:2258] T 105f3cff6c854e36ac35c3b82eb7b388 P 17a7225911ee45cab7c491dc1cbae7b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:18:28.471161 31932 tablet_replica.cc:330] T 105f3cff6c854e36ac35c3b82eb7b388 P 5acd4aea3b3a41a4810d08531f175532: stopping tablet replica
I20260929 03:18:28.471252 31932 raft_consensus.cc:2229] T 105f3cff6c854e36ac35c3b82eb7b388 P 5acd4aea3b3a41a4810d08531f175532 [term 1 LEADER]: Raft consensus shutting down.
I20260929 03:18:28.471356 31932 raft_consensus.cc:2258] T 105f3cff6c854e36ac35c3b82eb7b388 P 5acd4aea3b3a41a4810d08531f175532 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:18:28.471845 31932 ts_tablet_manager.cc:1905] T 105f3cff6c854e36ac35c3b82eb7b388 P 5acd4aea3b3a41a4810d08531f175532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:18:28.472350 31930 ts_tablet_manager.cc:1905] T 105f3cff6c854e36ac35c3b82eb7b388 P 17a7225911ee45cab7c491dc1cbae7b8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:18:28.473999 31932 ts_tablet_manager.cc:1918] T 105f3cff6c854e36ac35c3b82eb7b388 P 5acd4aea3b3a41a4810d08531f175532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:18:28.474126 31932 log.cc:1196] T 105f3cff6c854e36ac35c3b82eb7b388 P 5acd4aea3b3a41a4810d08531f175532: Deleting WAL directory at /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651852476208-21193-0/minicluster-data/ts-2/wal/wals/105f3cff6c854e36ac35c3b82eb7b388
I20260929 03:18:28.474283 31930 ts_tablet_manager.cc:1918] T 105f3cff6c854e36ac35c3b82eb7b388 P 17a7225911ee45cab7c491dc1cbae7b8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:18:28.474380 31932 ts_tablet_manager.cc:1939] T 105f3cff6c854e36ac35c3b82eb7b388 P 5acd4aea3b3a41a4810d08531f175532: Deleting consensus metadata
I20260929 03:18:28.474444 31930 log.cc:1196] T 105f3cff6c854e36ac35c3b82eb7b388 P 17a7225911ee45cab7c491dc1cbae7b8: Deleting WAL directory at /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651852476208-21193-0/minicluster-data/ts-1/wal/wals/105f3cff6c854e36ac35c3b82eb7b388
I20260929 03:18:28.474788 31930 ts_tablet_manager.cc:1939] T 105f3cff6c854e36ac35c3b82eb7b388 P 17a7225911ee45cab7c491dc1cbae7b8: Deleting consensus metadata
I20260929 03:18:28.474798 31420 catalog_manager.cc:4841] TS 5acd4aea3b3a41a4810d08531f175532 (127.20.178.67:45619): tablet 105f3cff6c854e36ac35c3b82eb7b388 (table test-table [id=05ed7055c156480d9df661cd62689db1]) successfully deleted
I20260929 03:18:28.475322 31421 catalog_manager.cc:4841] TS 17a7225911ee45cab7c491dc1cbae7b8 (127.20.178.66:43733): tablet 105f3cff6c854e36ac35c3b82eb7b388 (table test-table [id=05ed7055c156480d9df661cd62689db1]) successfully deleted
W20260929 03:18:28.479204 31422 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.20.178.65:43035 (error 108)
I20260929 03:18:28.479403 21193 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuoaHbX/build/release/bin/kudu with pid 31631
I20260929 03:18:28.480204 31902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:28.479583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.65:43035 (local address 127.0.0.1:60712)
0929 03:18:28.480114 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:28.480153 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.178.65:43035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":510}
W20260929 03:18:28.480355 31422 catalog_manager.cc:4584] TS 0832e6f8c5684bc5a0b08ff65668b631 (127.20.178.65:43035): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 105f3cff6c854e36ac35c3b82eb7b388: Network error: Client connection negotiation failed: client connection to 127.20.178.65:43035: connect: Connection refused (error 111)
I20260929 03:18:28.485435 21193 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuoaHbX/build/release/bin/kudu with pid 31765
I20260929 03:18:28.490057 21193 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuoaHbX/build/release/bin/kudu with pid 31403
2026-09-29T03:18:28Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (1832 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651852476208-21193-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:18:28 dist-test-slave-c9qh krb5kdc[31936](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:18:28 dist-test-slave-c9qh krb5kdc[31936](info): set up 2 sockets
Sep 29 03:18:28 dist-test-slave-c9qh krb5kdc[31936](info): commencing operation
krb5kdc: starting...
W20260929 03:18:29.517665 21193 mini_kdc.cc:121] Time spent starting KDC: real 1.008s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651852476208-21193-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651852476208-21193-0/krb5kdc/test-user.keytab.
Sep 29 03:18:29 dist-test-slave-c9qh krb5kdc[31936](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651909, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:18:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:18:29Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.20.178.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.20.178.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.20.178.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651852476208-21193-0/krb5kdc/rangeradmin_127.20.178.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.20.178.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651852476208-21193-0/krb5kdc/rangeradmin_127.20.178.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.20.178.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.20.178.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.20.178.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651852476208-21193-0/krb5kdc/rangerlookup_127.20.178.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.20.178.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651852476208-21193-0/krb5kdc/rangerlookup_127.20.178.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.20.178.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.20.178.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.178.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651852476208-21193-0/krb5kdc/HTTP_127.20.178.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.20.178.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651852476208-21193-0/krb5kdc/HTTP_127.20.178.84@KRBTEST.COM.keytab.
I20260929 03:18:29.675966 21193 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651852476208-21193-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskuoaHbX/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651852476208-21193-0/minicluster-data/postgres -l logfile start
2026-09-29 03:18:32.134 UTC [31966] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-29 03:18:32.135 UTC [31966] LOG: listening on IPv4 address "127.20.178.84", port 60899
2026-09-29 03:18:32.138 UTC [31966] LOG: listening on Unix socket "/tmp/.s.PGSQL.60899"
2026-09-29 03:18:32.171 UTC [31969] LOG: database system was shut down at 2026-09-29 03:18:30 UTC
2026-09-29 03:18:32.174 UTC [31966] LOG: database system is ready to accept connections
I20260929 03:18:33.117527 21193 mini_postgres.cc:96] Postgres bound to 60899
2026-09-29 03:18:33.123 UTC [31977] FATAL: database "slave" does not exist
127.20.178.84:60899 - accepting connections
I20260929 03:18:33.124100 21193 mini_ranger.cc:161] Starting Ranger...
I20260929 03:18:33.138152 21193 mini_ranger.cc:84] Created miniranger Postgres user
I20260929 03:18:33.612493 21193 mini_ranger.cc:87] Created ranger Postgres database
I20260929 03:18:33.612560 21193 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651852476208-21193-0/minicluster-data/ranger-admin
2026-09-29 03:18:33,876 [I] DB FLAVOR :POSTGRES
2026-09-29 03:18:33,877 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 03:18:33,877 [I] Checking connection..
2026-09-29 03:18:33,877 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuoaHbX/build/release/bin/postgresql.jar:/tmp/dist-test-taskuoaHbX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.178.84:60899/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-29 03:18:34,126 [I] Checking connection passed.
2026-09-29 03:18:34,126 [I] --------- Verifying version history table ---------
2026-09-29 03:18:34,127 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuoaHbX/build/release/bin/postgresql.jar:/tmp/dist-test-taskuoaHbX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.178.84:60899/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 03:18:34,381 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 03:18:34,382 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuoaHbX/build/release/bin/postgresql.jar:/tmp/dist-test-taskuoaHbX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.178.84:60899/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 03:18:34,608 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 03:18:34,608 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-29 03:18:34,609 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuoaHbX/build/release/bin/postgresql.jar:/tmp/dist-test-taskuoaHbX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.178.84:60899/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskuoaHbX/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-29 03:18:34.835 UTC [32041] WARNING: there is no transaction in progress
2026-09-29 03:18:34,841 [I] create_dbversion_catalog.sql file imported successfully
2026-09-29 03:18:34,842 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuoaHbX/build/release/bin/postgresql.jar:/tmp/dist-test-taskuoaHbX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.178.84:60899/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 03:18:35,070 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-29 03:18:35,071 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 03:18:35,071 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuoaHbX/build/release/bin/postgresql.jar:/tmp/dist-test-taskuoaHbX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.178.84:60899/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-29 03:18:35,301 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuoaHbX/build/release/bin/postgresql.jar:/tmp/dist-test-taskuoaHbX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.178.84:60899/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-29 03:18:35,533 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuoaHbX/build/release/bin/postgresql.jar:/tmp/dist-test-taskuoaHbX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.178.84:60899/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-c9qh.c.gcp-upstream.internal','N') ;"
2026-09-29 03:18:35,755 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuoaHbX/build/release/bin/postgresql.jar:/tmp/dist-test-taskuoaHbX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.178.84:60899/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 03:18:36,018 [I] Table x_portal_user does not exist in database ranger
2026-09-29 03:18:36,018 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuoaHbX/build/release/bin/postgresql.jar:/tmp/dist-test-taskuoaHbX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.178.84:60899/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 03:18:36,248 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-29 03:18:36,248 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuoaHbX/build/release/bin/postgresql.jar:/tmp/dist-test-taskuoaHbX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.178.84:60899/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 03:18:36,476 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuoaHbX/build/release/bin/postgresql.jar:/tmp/dist-test-taskuoaHbX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.178.84:60899/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 03:18:36,723 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-29 03:18:36,723 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuoaHbX/build/release/bin/postgresql.jar:/tmp/dist-test-taskuoaHbX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.178.84:60899/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskuoaHbX/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-29 03:18:37.697 UTC [32149] WARNING: there is no transaction in progress
2026-09-29 03:18:37.715 UTC [32149] WARNING: there is no transaction in progress
2026-09-29 03:18:37.730 UTC [32149] WARNING: there is no transaction in progress
2026-09-29 03:18:37.745 UTC [32149] WARNING: there is no transaction in progress
2026-09-29 03:18:37.760 UTC [32149] WARNING: there is no transaction in progress
2026-09-29 03:18:37.777 UTC [32149] WARNING: there is no transaction in progress
2026-09-29 03:18:37.838 UTC [32149] WARNING: there is no transaction in progress
2026-09-29 03:18:37.845 UTC [32149] WARNING: there is no transaction in progress
2026-09-29 03:18:37.860 UTC [32149] WARNING: there is no transaction in progress
2026-09-29 03:18:37.870 UTC [32149] WARNING: there is no transaction in progress
2026-09-29 03:18:37.880 UTC [32149] WARNING: there is no transaction in progress
2026-09-29 03:18:37.893 UTC [32149] WARNING: there is no transaction in progress
2026-09-29 03:18:37.902 UTC [32149] WARNING: there is no transaction in progress
2026-09-29 03:18:37.908 UTC [32149] WARNING: there is no transaction in progress
2026-09-29 03:18:37.915 UTC [32149] WARNING: there is no transaction in progress
2026-09-29 03:18:37.921 UTC [32149] WARNING: there is no transaction in progress
2026-09-29 03:18:38,548 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-29 03:18:38,548 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuoaHbX/build/release/bin/postgresql.jar:/tmp/dist-test-taskuoaHbX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.178.84:60899/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-29 03:18:38,792 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-29 03:18:38,793 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuoaHbX/build/release/bin/postgresql.jar:/tmp/dist-test-taskuoaHbX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.178.84:60899/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 03:18:39,021 [I] Table x_portal_user already exists in database 'ranger'
2026-09-29 03:18:39,022 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuoaHbX/build/release/bin/postgresql.jar:/tmp/dist-test-taskuoaHbX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.178.84:60899/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 03:18:39,263 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-29 03:18:39,263 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuoaHbX/build/release/bin/postgresql.jar:/tmp/dist-test-taskuoaHbX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.178.84:60899/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 03:18:39,500 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuoaHbX/build/release/bin/postgresql.jar:/tmp/dist-test-taskuoaHbX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.178.84:60899/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 03:18:39,738 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuoaHbX/build/release/bin/postgresql.jar:/tmp/dist-test-taskuoaHbX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.178.84:60899/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-c9qh.c.gcp-upstream.internal';"
2026-09-29 03:18:39,984 [I] CORE_DB_SCHEMA import status has been updated
2026-09-29 03:18:39,984 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuoaHbX/build/release/bin/postgresql.jar:/tmp/dist-test-taskuoaHbX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.178.84:60899/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-29 03:18:40,237 [I] DB_PATCHES have already been applied
I20260929 03:18:40.243077 21193 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651852476208-21193-0/minicluster-data/ranger-admin:/tmp/dist-test-taskuoaHbX/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskuoaHbX/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskuoaHbX/build/release/bin/postgresql.jar:/tmp/dist-test-taskuoaHbX/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260929 03:18:40.243139 21193 mini_ranger.cc:193] Using host: 127.20.178.84
I20260929 03:18:40.246126 21193 mini_ranger.cc:227] Ranger admin URL: http://127.20.178.84:45491
Sep 29, 2026 3:18:40 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 3:18:40 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 3:18:40 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651852476208-21193-0/minicluster-data/ranger-admin/ews/webapp
Sep 29, 2026 3:18:40 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651852476208-21193-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 29, 2026 3:18:40 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651852476208-21193-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 29, 2026 3:18:41 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651852476208-21193-0/minicluster-data/ranger-admin/ews/webapp].
Sep 29, 2026 3:18:41 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260929 03:18:41.335004 21193 mini_ranger.cc:160] Time spent starting Ranger: real 8.211s user 0.000s sys 0.009s
Sep 29, 2026 3:19:01 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 29, 2026 3:19:01 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.MetricsREST
Sep 29, 2026 3:19:01 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 29, 2026 3:19:01 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 29, 2026 3:19:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 29, 2026 3:19:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 29, 2026 3:19:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 29, 2026 3:19:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 29, 2026 3:19:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 29, 2026 3:19:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 29, 2026 3:19:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 29, 2026 3:19:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 29, 2026 3:19:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 29, 2026 3:19:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 29, 2026 3:19:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 29, 2026 3:19:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 29, 2026 3:19:01 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 29, 2026 3:19:02 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260929 03:19:02.854311 21193 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.20.178.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.20.178.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.20.178.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651852476208-21193-0/krb5kdc/rangerkms_127.20.178.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.20.178.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651852476208-21193-0/krb5kdc/rangerkms_127.20.178.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.20.178.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.20.178.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.178.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651852476208-21193-0/krb5kdc/HTTP_127.20.178.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.20.178.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651852476208-21193-0/krb5kdc/HTTP_127.20.178.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 29 03:19:02 dist-test-slave-c9qh krb5kdc[31936](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651942, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260929 03:19:02.940193 21193 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260929 03:19:02.949307 21193 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260929 03:19:03.842548 21193 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260929 03:19:03.842645 21193 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651852476208-21193-0/minicluster-data/ranger-kms
I20260929 03:19:03.842669 21193 mini_ranger_kms.cc:227] Using postgres at 127.20.178.84:60899
2026-09-29 03:19:04,009 [I] DB FLAVOR :POSTGRES
2026-09-29 03:19:04,009 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 03:19:04,009 [I] Checking connection
2026-09-29 03:19:04,010 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuoaHbX/build/release/bin/postgresql.jar:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651852476208-21193-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.178.84:60899/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-29 03:19:04,257 [I] connection success
2026-09-29 03:19:04,258 [I] --------- Verifying Ranger DB tables ---------
2026-09-29 03:19:04,258 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-29 03:19:04,258 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuoaHbX/build/release/bin/postgresql.jar:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651852476208-21193-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.178.84:60899/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-29 03:19:04,483 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-29 03:19:04,484 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 03:19:04,484 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-29 03:19:04,484 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuoaHbX/build/release/bin/postgresql.jar:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651852476208-21193-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.178.84:60899/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651852476208-21193-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-29 03:19:04,729 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260929 03:19:05.154171 21193 mini_ranger_kms.cc:310] Created kms service
I20260929 03:19:05.327723 21193 mini_ranger_kms.cc:326] Created kudu user
I20260929 03:19:05.371661 21193 mini_ranger_kms.cc:343] Created rangerkms user
Sep 29 03:19:05 dist-test-slave-c9qh krb5kdc[31936](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651942, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.20.178.84@KRBTEST.COM
I20260929 03:19:05.770483 21193 mini_ranger_kms.cc:384] Added ranger policy
I20260929 03:19:05.770596 21193 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651852476208-21193-0/minicluster-data/ranger-kms:/tmp/dist-test-taskuoaHbX/build/release/bin/postgresql.jar:/tmp/dist-test-taskuoaHbX/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskuoaHbX/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260929 03:19:05.770619 21193 mini_ranger_kms.cc:240] Using host: 127.20.178.84
I20260929 03:19:05.774310 21193 mini_ranger_kms.cc:276] Ranger KMS PID: 32329
I20260929 03:19:05.774426 21193 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.20.178.84:55955
Sep 29, 2026 3:19:06 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 3:19:06 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 3:19:06 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651852476208-21193-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 29, 2026 3:19:06 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651852476208-21193-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 29, 2026 3:19:07 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651852476208-21193-0/minicluster-data/ranger-kms/ews/webapp].
Sep 29, 2026 3:19:07 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260929 03:19:08.018833 21193 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.079s user 0.004s sys 0.147s
I20260929 03:19:08.018945 21193 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260929 03:19:08.019032 21193 mini_ranger_kms.cc:401] 127.20.178.84:55955/kms/v1/keys
Sep 29 03:19:08 dist-test-slave-c9qh krb5kdc[31936](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790651948, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.20.178.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 29 03:19:08 dist-test-slave-c9qh krb5kdc[31936](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651948, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.20.178.84@KRBTEST.COM for HTTP/127.20.178.84@KRBTEST.COM
Sep 29 03:19:08 dist-test-slave-c9qh krb5kdc[31936](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651948, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.20.178.84@KRBTEST.COM for HTTP/127.20.178.84@KRBTEST.COM
Sep 29 03:19:11 dist-test-slave-c9qh krb5kdc[31936](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651951, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.20.178.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.178.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.178.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.126.keytab.
Entry for principal kudu/127.20.178.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.126.keytab.
I20260929 03:19:11.085628 21193 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuoaHbX/build/release/bin/kudu
/tmp/dist-test-taskuoaHbX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651852476208-21193-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651852476208-21193-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651852476208-21193-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651852476208-21193-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.178.126:45575
--webserver_interface=127.20.178.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.126.keytab
--principal=kudu/127.20.178.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.178.84:44217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.178.126:45575
--ranger_config_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651852476208-21193-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.20.178.84:55955/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651852476208-21193-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651852476208-21193-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651852476208-21193-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:19:11.186715 32380 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:19:11.186932 32380 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:19:11.186968 32380 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:19:11.188091 32380 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:19:11.188155 32380 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:19:11.188181 32380 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:19:11.188213 32380 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:19:11.188236 32380 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:19:11.189806 32380 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.178.84:44217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651852476208-21193-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651852476208-21193-0/minicluster-data/master-0/wal
--ranger_kms_url=127.20.178.84:55955/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.20.178.126:45575
--ranger_config_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651852476208-21193-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.126.keytab
--principal=kudu/127.20.178.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.178.126:45575
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651852476208-21193-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.178.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32380
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651852476208-21193-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:19:11.190069 32380 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:19:11.190291 32380 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:19:11.196410 32386 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:19:11.196419 32385 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:19:11.196453 32380 server_base.cc:1034] running on GCE node
W20260929 03:19:11.196735 32388 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:19:11.196997 32380 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:19:11.197315 32380 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:19:11.198488 32380 hybrid_clock.cc:648] HybridClock initialized: now 1790651951198476 us; error 29 us; skew 500 ppm
Sep 29 03:19:11 dist-test-slave-c9qh krb5kdc[31936](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651951, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:19:11.201411 32380 init.cc:377] Logged in from keytab as kudu/127.20.178.126@KRBTEST.COM (short username kudu)
I20260929 03:19:11.201835 32380 webserver.cc:460] Webserver started at http://127.20.178.126:40735/ using document root <none> and password file <none>
I20260929 03:19:11.202068 32380 fs_manager.cc:359] Metadata directory not provided
I20260929 03:19:11.202149 32380 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:19:11.202320 32380 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 03:19:11 dist-test-slave-c9qh krb5kdc[31936](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651951, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.126@KRBTEST.COM for HTTP/127.20.178.84@KRBTEST.COM
I20260929 03:19:11.247419 32380 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651852476208-21193-0/minicluster-data/master-0/data/instance:
uuid: "503c3193be3949c7a4233f5f67be9568"
format_stamp: "Formatted at 2026-09-29 03:19:11 on dist-test-slave-c9qh"
server_key: "5d02456e682f564e56bb7ff0b181ec7d"
server_key_iv: "ab006407aa49c1523de2ddec2a0ed48c"
server_key_version: "kuduclusterkey@0"
I20260929 03:19:11.247879 32380 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651852476208-21193-0/minicluster-data/master-0/wal/instance:
uuid: "503c3193be3949c7a4233f5f67be9568"
format_stamp: "Formatted at 2026-09-29 03:19:11 on dist-test-slave-c9qh"
server_key: "5d02456e682f564e56bb7ff0b181ec7d"
server_key_iv: "ab006407aa49c1523de2ddec2a0ed48c"
server_key_version: "kuduclusterkey@0"
I20260929 03:19:11.249787 32380 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260929 03:19:11.265815 32395 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:19:11.266067 32380 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 03:19:11.266178 32380 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651852476208-21193-0/minicluster-data/master-0/data,/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651852476208-21193-0/minicluster-data/master-0/wal
uuid: "503c3193be3949c7a4233f5f67be9568"
format_stamp: "Formatted at 2026-09-29 03:19:11 on dist-test-slave-c9qh"
server_key: "5d02456e682f564e56bb7ff0b181ec7d"
server_key_iv: "ab006407aa49c1523de2ddec2a0ed48c"
server_key_version: "kuduclusterkey@0"
I20260929 03:19:11.266276 32380 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651852476208-21193-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651852476208-21193-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651852476208-21193-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:19:11.288681 32380 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:19:11.290753 32380 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:19:11.290930 32380 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:19:11.296011 32380 rpc_server.cc:307] RPC server started. Bound to: 127.20.178.126:45575
I20260929 03:19:11.296037 32447 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.178.126:45575 every 8 connection(s)
I20260929 03:19:11.296452 32380 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651852476208-21193-0/minicluster-data/master-0/data/info.pb
I20260929 03:19:11.297200 32448 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:19:11.299398 32448 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 503c3193be3949c7a4233f5f67be9568: Bootstrap starting.
I20260929 03:19:11.299966 32448 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 503c3193be3949c7a4233f5f67be9568: Neither blocks nor log segments found. Creating new log.
I20260929 03:19:11.300248 32448 log.cc:824] T 00000000000000000000000000000000 P 503c3193be3949c7a4233f5f67be9568: Log is configured to *not* fsync() on all Append() calls
I20260929 03:19:11.300941 32448 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 503c3193be3949c7a4233f5f67be9568: No bootstrap required, opened a new log
I20260929 03:19:11.302143 21193 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuoaHbX/build/release/bin/kudu as pid 32380
I20260929 03:19:11.302270 21193 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651852476208-21193-0/minicluster-data/master-0/wal/instance
I20260929 03:19:11.302508 21193 external_mini_cluster.cc:1428] Setting key 77286f4442057c647c9155da9babc657
I20260929 03:19:11.303548 32448 raft_consensus.cc:348] T 00000000000000000000000000000000 P 503c3193be3949c7a4233f5f67be9568 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "503c3193be3949c7a4233f5f67be9568" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 45575 } }
I20260929 03:19:11.303787 32448 raft_consensus.cc:374] T 00000000000000000000000000000000 P 503c3193be3949c7a4233f5f67be9568 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:19:11.303844 32448 raft_consensus.cc:729] T 00000000000000000000000000000000 P 503c3193be3949c7a4233f5f67be9568 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 503c3193be3949c7a4233f5f67be9568, State: Initialized, Role: FOLLOWER
I20260929 03:19:11.303987 32448 consensus_queue.cc:260] T 00000000000000000000000000000000 P 503c3193be3949c7a4233f5f67be9568 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "503c3193be3949c7a4233f5f67be9568" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 45575 } }
I20260929 03:19:11.304091 32448 raft_consensus.cc:388] T 00000000000000000000000000000000 P 503c3193be3949c7a4233f5f67be9568 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:19:11.304144 32448 raft_consensus.cc:482] T 00000000000000000000000000000000 P 503c3193be3949c7a4233f5f67be9568 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:19:11.304214 32448 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 503c3193be3949c7a4233f5f67be9568 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:19:11.304898 32448 raft_consensus.cc:504] T 00000000000000000000000000000000 P 503c3193be3949c7a4233f5f67be9568 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "503c3193be3949c7a4233f5f67be9568" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 45575 } }
I20260929 03:19:11.305001 32448 leader_election.cc:302] T 00000000000000000000000000000000 P 503c3193be3949c7a4233f5f67be9568 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 503c3193be3949c7a4233f5f67be9568; no voters:
I20260929 03:19:11.305199 32448 leader_election.cc:288] T 00000000000000000000000000000000 P 503c3193be3949c7a4233f5f67be9568 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:19:11.305256 32453 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 503c3193be3949c7a4233f5f67be9568 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:19:11.305469 32453 raft_consensus.cc:686] T 00000000000000000000000000000000 P 503c3193be3949c7a4233f5f67be9568 [term 1 LEADER]: Becoming Leader. State: Replica: 503c3193be3949c7a4233f5f67be9568, State: Running, Role: LEADER
I20260929 03:19:11.305516 32448 sys_catalog.cc:564] T 00000000000000000000000000000000 P 503c3193be3949c7a4233f5f67be9568 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:19:11.305634 32453 consensus_queue.cc:237] T 00000000000000000000000000000000 P 503c3193be3949c7a4233f5f67be9568 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "503c3193be3949c7a4233f5f67be9568" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 45575 } }
I20260929 03:19:11.306049 32453 sys_catalog.cc:455] T 00000000000000000000000000000000 P 503c3193be3949c7a4233f5f67be9568 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "503c3193be3949c7a4233f5f67be9568" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "503c3193be3949c7a4233f5f67be9568" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 45575 } } }
I20260929 03:19:11.306926 32448 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651852476208-21193-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260929 03:19:11.306990 32454 sys_catalog.cc:455] T 00000000000000000000000000000000 P 503c3193be3949c7a4233f5f67be9568 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 503c3193be3949c7a4233f5f67be9568. Latest consensus state: current_term: 1 leader_uuid: "503c3193be3949c7a4233f5f67be9568" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "503c3193be3949c7a4233f5f67be9568" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 45575 } } }
I20260929 03:19:11.307236 32454 sys_catalog.cc:458] T 00000000000000000000000000000000 P 503c3193be3949c7a4233f5f67be9568 [sys.catalog]: This master's current role is: LEADER
I20260929 03:19:11.307406 32453 sys_catalog.cc:458] T 00000000000000000000000000000000 P 503c3193be3949c7a4233f5f67be9568 [sys.catalog]: This master's current role is: LEADER
I20260929 03:19:11.311136 32467 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:19:11.311734 32467 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 29 03:19:11 dist-test-slave-c9qh krb5kdc[31936](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651951, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.178.126@KRBTEST.COM
I20260929 03:19:11.316424 32467 catalog_manager.cc:1347] Generated new cluster ID: af74d164fc9f4aaa9758649fd4da87d9
I20260929 03:19:11.316476 32467 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:19:11.317241 32452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:11.303100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32898 (local address 127.20.178.126:45575)
0929 03:19:11.303262 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0929 03:19:11.303271 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 03:19:11.303312 (+ 41us) server_negotiation.cc:407] Connection header received
0929 03:19:11.303516 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:19:11.303523 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:19:11.303582 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:19:11.303726 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:19:11.304584 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:11.306058 (+ 1474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:11.307076 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:11.309891 (+ 2815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:11.310084 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:11.313287 (+ 3203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:19:11.313292 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:19:11.313292 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:19:11.313304 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:19:11.315319 (+ 2015us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:19:11.315830 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:19:11.315832 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:19:11.315833 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:19:11.315925 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:19:11.316140 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:19:11.316141 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:19:11.316141 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:19:11.316444 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:19:11.316583 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0929 03:19:11.316895 (+ 312us) server_negotiation.cc:299] Negotiation successful
0929 03:19:11.317025 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":89,"threads_started":1}
I20260929 03:19:11.349547 32467 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:19:11.350469 32467 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:19:11.353879 32467 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 503c3193be3949c7a4233f5f67be9568: Generated new TSK 0
I20260929 03:19:11.354140 32467 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.178.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.178.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.178.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.65.keytab.
Entry for principal kudu/127.20.178.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.65.keytab.
I20260929 03:19:12.120069 21193 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuoaHbX/build/release/bin/kudu
/tmp/dist-test-taskuoaHbX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651852476208-21193-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651852476208-21193-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651852476208-21193-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651852476208-21193-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.20.178.84:55955/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.178.65:0
--local_ip_for_outbound_sockets=127.20.178.65
--webserver_interface=127.20.178.65
--webserver_port=0
--tserver_master_addrs=127.20.178.126:45575
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.65.keytab
--principal=kudu/127.20.178.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.178.84:44217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651852476208-21193-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651852476208-21193-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651852476208-21193-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:19:12.224417 32493 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:19:12.224651 32493 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:19:12.224716 32493 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:19:12.225965 32493 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:19:12.226025 32493 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:19:12.226071 32493 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.178.65
I20260929 03:19:12.228207 32493 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.178.84:44217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651852476208-21193-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651852476208-21193-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.20.178.84:55955/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.65.keytab
--principal=kudu/127.20.178.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.178.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651852476208-21193-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.178.65
--webserver_port=0
--tserver_master_addrs=127.20.178.126:45575
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32493
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.178.65
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651852476208-21193-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:19:12.228600 32493 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:19:12.228928 32493 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:19:12.235399 32499 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:19:12.235417 32498 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:19:12.235419 32501 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:19:12.236394 32493 server_base.cc:1034] running on GCE node
I20260929 03:19:12.236670 32493 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:19:12.237007 32493 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:19:12.238175 32493 hybrid_clock.cc:648] HybridClock initialized: now 1790651952238155 us; error 33 us; skew 500 ppm
Sep 29 03:19:12 dist-test-slave-c9qh krb5kdc[31936](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651952, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:19:12.242583 32493 init.cc:377] Logged in from keytab as kudu/127.20.178.65@KRBTEST.COM (short username kudu)
I20260929 03:19:12.243076 32493 webserver.cc:460] Webserver started at http://127.20.178.65:43257/ using document root <none> and password file <none>
I20260929 03:19:12.243450 32493 fs_manager.cc:359] Metadata directory not provided
I20260929 03:19:12.243564 32493 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:19:12.243750 32493 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 03:19:12 dist-test-slave-c9qh krb5kdc[31936](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651952, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.65@KRBTEST.COM for HTTP/127.20.178.84@KRBTEST.COM
I20260929 03:19:12.256369 32493 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651852476208-21193-0/minicluster-data/ts-0/data/instance:
uuid: "60fde20ef0914a439682ff4badecf345"
format_stamp: "Formatted at 2026-09-29 03:19:12 on dist-test-slave-c9qh"
server_key: "3dc2ee66eb96ea396e92d52f69f177d8"
server_key_iv: "499b3ad631ff195733fea1573df06ba2"
server_key_version: "kuduclusterkey@0"
I20260929 03:19:12.256888 32493 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651852476208-21193-0/minicluster-data/ts-0/wal/instance:
uuid: "60fde20ef0914a439682ff4badecf345"
format_stamp: "Formatted at 2026-09-29 03:19:12 on dist-test-slave-c9qh"
server_key: "3dc2ee66eb96ea396e92d52f69f177d8"
server_key_iv: "499b3ad631ff195733fea1573df06ba2"
server_key_version: "kuduclusterkey@0"
I20260929 03:19:12.258598 32493 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260929 03:19:12.270188 32508 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:19:12.270599 32493 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 03:19:12.270694 32493 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651852476208-21193-0/minicluster-data/ts-0/data,/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651852476208-21193-0/minicluster-data/ts-0/wal
uuid: "60fde20ef0914a439682ff4badecf345"
format_stamp: "Formatted at 2026-09-29 03:19:12 on dist-test-slave-c9qh"
server_key: "3dc2ee66eb96ea396e92d52f69f177d8"
server_key_iv: "499b3ad631ff195733fea1573df06ba2"
server_key_version: "kuduclusterkey@0"
I20260929 03:19:12.270802 32493 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651852476208-21193-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651852476208-21193-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651852476208-21193-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:19:12.303289 32493 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:19:12.305806 32493 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:19:12.305959 32493 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:19:12.306218 32493 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:19:12.306667 32493 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:19:12.306720 32493 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:19:12.306751 32493 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:19:12.306790 32493 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:19:12.314548 32493 rpc_server.cc:307] RPC server started. Bound to: 127.20.178.65:38029
I20260929 03:19:12.314973 32493 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651852476208-21193-0/minicluster-data/ts-0/data/info.pb
I20260929 03:19:12.315466 32623 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.178.65:38029 every 8 connection(s)
I20260929 03:19:12.317000 21193 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuoaHbX/build/release/bin/kudu as pid 32493
I20260929 03:19:12.317097 21193 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651852476208-21193-0/minicluster-data/ts-0/wal/instance
I20260929 03:19:12.317309 21193 external_mini_cluster.cc:1428] Setting key 17e8c44cc1bcc01344b8ff0543db5df2
Sep 29 03:19:12 dist-test-slave-c9qh krb5kdc[31936](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651952, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.65@KRBTEST.COM for kudu/127.20.178.126@KRBTEST.COM
I20260929 03:19:12.326476 32626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:12.316180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:45575 (local address 127.20.178.65:32935)
0929 03:19:12.316343 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0929 03:19:12.316385 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 03:19:12.316876 (+ 491us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:19:12.317152 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:19:12.317158 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:19:12.317344 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:19:12.317622 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:19:12.317630 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:19:12.318741 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:19:12.318742 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:19:12.320031 (+ 1289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:19:12.320034 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:19:12.320572 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:19:12.320573 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:19:12.320754 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:12.321373 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:19:12.321383 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:19:12.322968 (+ 1585us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:19:12.324936 (+ 1968us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:19:12.324942 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:19:12.324944 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:19:12.325214 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:19:12.325426 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:19:12.325427 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:19:12.325427 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:19:12.325545 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:19:12.325821 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:19:12.325827 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:19:12.325974 (+ 147us) client_negotiation.cc:769] Sending connection context
0929 03:19:12.326087 (+ 113us) client_negotiation.cc:240] Negotiation successful
0929 03:19:12.326248 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":94,"threads_started":1}
I20260929 03:19:12.326900 32627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:12.316227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.65:32935 (local address 127.20.178.126:45575)
0929 03:19:12.316389 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0929 03:19:12.316392 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:19:12.316489 (+ 97us) server_negotiation.cc:407] Connection header received
0929 03:19:12.316986 (+ 497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:19:12.316987 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:19:12.317002 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:19:12.317079 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:19:12.317918 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:12.318630 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:12.320133 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:12.320438 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:12.320505 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:12.323071 (+ 2566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:19:12.323072 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:19:12.323073 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:19:12.323080 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:19:12.324848 (+ 1768us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:19:12.325299 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:19:12.325300 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:19:12.325300 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:19:12.325361 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:19:12.325607 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:19:12.325608 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:19:12.325608 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:19:12.325728 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:19:12.325796 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0929 03:19:12.326712 (+ 916us) server_negotiation.cc:299] Negotiation successful
0929 03:19:12.326805 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":85,"threads_started":1}
WARNING: no policy specified for kudu/127.20.178.66@KRBTEST.COM; defaulting to no policy
I20260929 03:19:12.327677 32624 heartbeater.cc:347] Connected to a master server at 127.20.178.126:45575
I20260929 03:19:12.327795 32624 heartbeater.cc:464] Registering TS with master...
I20260929 03:19:12.327973 32624 heartbeater.cc:511] Master 127.20.178.126:45575 requested a full tablet report, sending...
I20260929 03:19:12.328466 32412 ts_manager.cc:194] Registered new tserver with Master: 60fde20ef0914a439682ff4badecf345 (127.20.178.65:38029)
I20260929 03:19:12.329334 32412 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.178.65@KRBTEST.COM'} at 127.20.178.65:32935
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.178.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.178.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.66.keytab.
Entry for principal kudu/127.20.178.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.66.keytab.
I20260929 03:19:12.349864 21193 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuoaHbX/build/release/bin/kudu
/tmp/dist-test-taskuoaHbX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651852476208-21193-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651852476208-21193-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651852476208-21193-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651852476208-21193-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.20.178.84:55955/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.178.66:0
--local_ip_for_outbound_sockets=127.20.178.66
--webserver_interface=127.20.178.66
--webserver_port=0
--tserver_master_addrs=127.20.178.126:45575
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.66.keytab
--principal=kudu/127.20.178.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.178.84:44217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651852476208-21193-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651852476208-21193-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651852476208-21193-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:19:12.450541 32630 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:19:12.450758 32630 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:19:12.450796 32630 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:19:12.452024 32630 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:19:12.452090 32630 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:19:12.452133 32630 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.178.66
I20260929 03:19:12.453613 32630 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.178.84:44217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651852476208-21193-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651852476208-21193-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.20.178.84:55955/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.66.keytab
--principal=kudu/127.20.178.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.178.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651852476208-21193-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.178.66
--webserver_port=0
--tserver_master_addrs=127.20.178.126:45575
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32630
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.178.66
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651852476208-21193-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:19:12.453867 32630 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:19:12.454069 32630 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:19:12.459137 32638 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:19:12.459456 32640 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:19:12.459561 32637 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:19:12.459839 32630 server_base.cc:1034] running on GCE node
I20260929 03:19:12.460114 32630 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:19:12.460378 32630 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:19:12.461545 32630 hybrid_clock.cc:648] HybridClock initialized: now 1790651952461523 us; error 39 us; skew 500 ppm
Sep 29 03:19:12 dist-test-slave-c9qh krb5kdc[31936](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651952, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:19:12.464125 32630 init.cc:377] Logged in from keytab as kudu/127.20.178.66@KRBTEST.COM (short username kudu)
I20260929 03:19:12.464620 32630 webserver.cc:460] Webserver started at http://127.20.178.66:44217/ using document root <none> and password file <none>
I20260929 03:19:12.464939 32630 fs_manager.cc:359] Metadata directory not provided
I20260929 03:19:12.465004 32630 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:19:12.465126 32630 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 03:19:12 dist-test-slave-c9qh krb5kdc[31936](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651952, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.66@KRBTEST.COM for HTTP/127.20.178.84@KRBTEST.COM
Sep 29 03:19:12 dist-test-slave-c9qh krb5kdc[31936](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790651952, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:19:12.486321 32630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651852476208-21193-0/minicluster-data/ts-1/data/instance:
uuid: "db3b3ccf47784156b2a0a3423ebec2d2"
format_stamp: "Formatted at 2026-09-29 03:19:12 on dist-test-slave-c9qh"
server_key: "57134f80a74e90441955ecd5854699f7"
server_key_iv: "42c2a98aa5350efe9a4ec7ff241b0ff6"
server_key_version: "kuduclusterkey@0"
I20260929 03:19:12.486729 32630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651852476208-21193-0/minicluster-data/ts-1/wal/instance:
uuid: "db3b3ccf47784156b2a0a3423ebec2d2"
format_stamp: "Formatted at 2026-09-29 03:19:12 on dist-test-slave-c9qh"
server_key: "57134f80a74e90441955ecd5854699f7"
server_key_iv: "42c2a98aa5350efe9a4ec7ff241b0ff6"
server_key_version: "kuduclusterkey@0"
I20260929 03:19:12.493052 32630 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.000s
I20260929 03:19:12.514955 32647 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:19:12.515573 32630 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.001s sys 0.000s
I20260929 03:19:12.515686 32630 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651852476208-21193-0/minicluster-data/ts-1/data,/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651852476208-21193-0/minicluster-data/ts-1/wal
uuid: "db3b3ccf47784156b2a0a3423ebec2d2"
format_stamp: "Formatted at 2026-09-29 03:19:12 on dist-test-slave-c9qh"
server_key: "57134f80a74e90441955ecd5854699f7"
server_key_iv: "42c2a98aa5350efe9a4ec7ff241b0ff6"
server_key_version: "kuduclusterkey@0"
I20260929 03:19:12.515799 32630 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651852476208-21193-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651852476208-21193-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651852476208-21193-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:19:12.545666 32630 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:19:12.548184 32630 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:19:12.548405 32630 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:19:12.548704 32630 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:19:12.549142 32630 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:19:12.549233 32630 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:19:12.549324 32630 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:19:12.549388 32630 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:19:12.556010 32630 rpc_server.cc:307] RPC server started. Bound to: 127.20.178.66:33431
I20260929 03:19:12.556725 32630 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651852476208-21193-0/minicluster-data/ts-1/data/info.pb
I20260929 03:19:12.558935 21193 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuoaHbX/build/release/bin/kudu as pid 32630
I20260929 03:19:12.559036 21193 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651852476208-21193-0/minicluster-data/ts-1/wal/instance
I20260929 03:19:12.559212 21193 external_mini_cluster.cc:1428] Setting key 7d3965aa8d64ba6e337fc6ffaf6cb3dd
I20260929 03:19:12.559288 32761 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.178.66:33431 every 8 connection(s)
Sep 29 03:19:12 dist-test-slave-c9qh krb5kdc[31936](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651952, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.66@KRBTEST.COM for kudu/127.20.178.126@KRBTEST.COM
I20260929 03:19:12.571337 32627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:12.560293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.66:60501 (local address 127.20.178.126:45575)
0929 03:19:12.560344 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0929 03:19:12.560346 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:19:12.561047 (+ 701us) server_negotiation.cc:407] Connection header received
0929 03:19:12.561700 (+ 653us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:19:12.561701 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:19:12.561717 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:19:12.561748 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:19:12.562579 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:12.563361 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:12.564910 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:12.565235 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:12.565282 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:12.567876 (+ 2594us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:19:12.567878 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:19:12.567878 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:19:12.567885 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:19:12.569651 (+ 1766us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:19:12.570144 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:19:12.570145 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:19:12.570145 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:19:12.570208 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:19:12.570548 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:19:12.570549 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:19:12.570549 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:19:12.570680 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:19:12.570726 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0929 03:19:12.571088 (+ 362us) server_negotiation.cc:299] Negotiation successful
0929 03:19:12.571178 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260929 03:19:12.572259 32764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:12.560019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:45575 (local address 127.20.178.66:60501)
0929 03:19:12.560922 (+ 903us) negotiation.cc:107] Waiting for socket to connect
0929 03:19:12.560970 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0929 03:19:12.561587 (+ 617us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:19:12.561938 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:19:12.561943 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:19:12.562174 (+ 231us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:19:12.562469 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:19:12.562476 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:19:12.563481 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:19:12.563482 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:19:12.564812 (+ 1330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:19:12.564815 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:19:12.565373 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:19:12.565374 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:19:12.565510 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:12.566117 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:19:12.566127 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:19:12.567769 (+ 1642us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:19:12.569748 (+ 1979us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:19:12.569754 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:19:12.569756 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:19:12.570052 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:19:12.570297 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:19:12.570298 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:19:12.570298 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:19:12.570472 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:19:12.570777 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:19:12.570783 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:19:12.570939 (+ 156us) client_negotiation.cc:769] Sending connection context
0929 03:19:12.571786 (+ 847us) client_negotiation.cc:240] Negotiation successful
0929 03:19:12.571954 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":862,"thread_start_us":82,"threads_started":1}
I20260929 03:19:12.572994 32762 heartbeater.cc:347] Connected to a master server at 127.20.178.126:45575
I20260929 03:19:12.573088 32762 heartbeater.cc:464] Registering TS with master...
I20260929 03:19:12.573290 32762 heartbeater.cc:511] Master 127.20.178.126:45575 requested a full tablet report, sending...
I20260929 03:19:12.573736 32412 ts_manager.cc:194] Registered new tserver with Master: db3b3ccf47784156b2a0a3423ebec2d2 (127.20.178.66:33431)
I20260929 03:19:12.574234 32412 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.178.66@KRBTEST.COM'} at 127.20.178.66:60501
WARNING: no policy specified for kudu/127.20.178.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.178.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.178.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.67.keytab.
Entry for principal kudu/127.20.178.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.67.keytab.
I20260929 03:19:12.597749 21193 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuoaHbX/build/release/bin/kudu
/tmp/dist-test-taskuoaHbX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651852476208-21193-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651852476208-21193-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651852476208-21193-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651852476208-21193-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.20.178.84:55955/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.178.67:0
--local_ip_for_outbound_sockets=127.20.178.67
--webserver_interface=127.20.178.67
--webserver_port=0
--tserver_master_addrs=127.20.178.126:45575
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.67.keytab
--principal=kudu/127.20.178.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.178.84:44217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651852476208-21193-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651852476208-21193-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651852476208-21193-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:19:12.717667 32767 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:19:12.717958 32767 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:19:12.717998 32767 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:19:12.719663 32767 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:19:12.719726 32767 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:19:12.719767 32767 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.178.67
I20260929 03:19:12.722077 32767 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.178.84:44217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651852476208-21193-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651852476208-21193-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.20.178.84:55955/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.67.keytab
--principal=kudu/127.20.178.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.178.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651852476208-21193-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.178.67
--webserver_port=0
--tserver_master_addrs=127.20.178.126:45575
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32767
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.178.67
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651852476208-21193-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:19:12.722340 32767 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:19:12.722600 32767 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:19:12.735116 305 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:19:12.735116 307 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:19:12.735653 304 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:19:12.736296 32767 server_base.cc:1034] running on GCE node
I20260929 03:19:12.736668 32767 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:19:12.736956 32767 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:19:12.738307 32767 hybrid_clock.cc:648] HybridClock initialized: now 1790651952738286 us; error 36 us; skew 500 ppm
Sep 29 03:19:12 dist-test-slave-c9qh krb5kdc[31936](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651952, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:19:12.741937 32767 init.cc:377] Logged in from keytab as kudu/127.20.178.67@KRBTEST.COM (short username kudu)
I20260929 03:19:12.742519 32767 webserver.cc:460] Webserver started at http://127.20.178.67:41531/ using document root <none> and password file <none>
I20260929 03:19:12.742818 32767 fs_manager.cc:359] Metadata directory not provided
I20260929 03:19:12.742885 32767 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:19:12.743017 32767 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 03:19:12 dist-test-slave-c9qh krb5kdc[31936](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651952, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.67@KRBTEST.COM for HTTP/127.20.178.84@KRBTEST.COM
I20260929 03:19:12.755891 32767 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651852476208-21193-0/minicluster-data/ts-2/data/instance:
uuid: "369a835dde204fb182fc44de821a7f15"
format_stamp: "Formatted at 2026-09-29 03:19:12 on dist-test-slave-c9qh"
server_key: "7d4717a41c17ec3e98da62c18ef9395d"
server_key_iv: "db6ff084ee53b2321db4e3778473c249"
server_key_version: "kuduclusterkey@0"
I20260929 03:19:12.756409 32767 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651852476208-21193-0/minicluster-data/ts-2/wal/instance:
uuid: "369a835dde204fb182fc44de821a7f15"
format_stamp: "Formatted at 2026-09-29 03:19:12 on dist-test-slave-c9qh"
server_key: "7d4717a41c17ec3e98da62c18ef9395d"
server_key_iv: "db6ff084ee53b2321db4e3778473c249"
server_key_version: "kuduclusterkey@0"
I20260929 03:19:12.759562 32767 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.000s
I20260929 03:19:12.776767 314 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:19:12.776968 32767 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.000s sys 0.001s
I20260929 03:19:12.777074 32767 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651852476208-21193-0/minicluster-data/ts-2/data,/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651852476208-21193-0/minicluster-data/ts-2/wal
uuid: "369a835dde204fb182fc44de821a7f15"
format_stamp: "Formatted at 2026-09-29 03:19:12 on dist-test-slave-c9qh"
server_key: "7d4717a41c17ec3e98da62c18ef9395d"
server_key_iv: "db6ff084ee53b2321db4e3778473c249"
server_key_version: "kuduclusterkey@0"
I20260929 03:19:12.777179 32767 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651852476208-21193-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651852476208-21193-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651852476208-21193-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:19:12.812927 32767 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:19:12.815244 32767 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:19:12.815400 32767 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:19:12.815623 32767 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:19:12.816010 32767 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:19:12.816063 32767 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:19:12.816102 32767 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:19:12.816128 32767 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:19:12.822911 32767 rpc_server.cc:307] RPC server started. Bound to: 127.20.178.67:43481
I20260929 03:19:12.823349 32767 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651852476208-21193-0/minicluster-data/ts-2/data/info.pb
I20260929 03:19:12.825222 427 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.178.67:43481 every 8 connection(s)
I20260929 03:19:12.831164 21193 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuoaHbX/build/release/bin/kudu as pid 32767
I20260929 03:19:12.831269 21193 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651852476208-21193-0/minicluster-data/ts-2/wal/instance
I20260929 03:19:12.831446 21193 external_mini_cluster.cc:1428] Setting key 576d3d8e363dc614b2f048eba4d31377
Sep 29 03:19:12 dist-test-slave-c9qh krb5kdc[31936](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651952, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.67@KRBTEST.COM for kudu/127.20.178.126@KRBTEST.COM
I20260929 03:19:12.838513 32627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:12.826439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.67:58345 (local address 127.20.178.126:45575)
0929 03:19:12.826487 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0929 03:19:12.826489 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:19:12.826727 (+ 238us) server_negotiation.cc:407] Connection header received
0929 03:19:12.827426 (+ 699us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:19:12.827428 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:19:12.827440 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:19:12.827472 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:19:12.828298 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:12.829074 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:12.830564 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:12.830861 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:12.830909 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:12.835296 (+ 4387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:19:12.835297 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:19:12.835298 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:19:12.835305 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:19:12.836984 (+ 1679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:19:12.837474 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:19:12.837476 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:19:12.837476 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:19:12.837533 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:19:12.837796 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:19:12.837796 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:19:12.837797 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:19:12.837924 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:19:12.837965 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0929 03:19:12.838321 (+ 356us) server_negotiation.cc:299] Negotiation successful
0929 03:19:12.838413 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260929 03:19:12.839260 430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:12.826447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:45575 (local address 127.20.178.67:58345)
0929 03:19:12.826601 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0929 03:19:12.826644 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 03:19:12.827276 (+ 632us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:19:12.827660 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:19:12.827664 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:19:12.827905 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:19:12.828184 (+ 279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:19:12.828190 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:19:12.829191 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:19:12.829192 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:19:12.830454 (+ 1262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:19:12.830457 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:19:12.831991 (+ 1534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:19:12.831992 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:19:12.832458 (+ 466us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:12.833123 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:19:12.833133 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:19:12.835182 (+ 2049us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:19:12.837078 (+ 1896us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:19:12.837085 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:19:12.837087 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:19:12.837380 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:19:12.837600 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:19:12.837600 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:19:12.837601 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:19:12.837725 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:19:12.838008 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:19:12.838014 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:19:12.838170 (+ 156us) client_negotiation.cc:769] Sending connection context
0929 03:19:12.838872 (+ 702us) client_negotiation.cc:240] Negotiation successful
0929 03:19:12.839030 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":80,"threads_started":1}
I20260929 03:19:12.840065 428 heartbeater.cc:347] Connected to a master server at 127.20.178.126:45575
I20260929 03:19:12.840193 428 heartbeater.cc:464] Registering TS with master...
I20260929 03:19:12.840397 428 heartbeater.cc:511] Master 127.20.178.126:45575 requested a full tablet report, sending...
I20260929 03:19:12.840847 32412 ts_manager.cc:194] Registered new tserver with Master: 369a835dde204fb182fc44de821a7f15 (127.20.178.67:43481)
I20260929 03:19:12.841313 32412 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.178.67@KRBTEST.COM'} at 127.20.178.67:58345
I20260929 03:19:12.843029 21193 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 03:19:12.867633 32627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:12.848932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32900 (local address 127.20.178.126:45575)
0929 03:19:12.848984 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0929 03:19:12.848986 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:19:12.856812 (+ 7826us) server_negotiation.cc:407] Connection header received
0929 03:19:12.856902 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:19:12.856903 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:19:12.856915 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:19:12.856964 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:19:12.860750 (+ 3786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:12.861513 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:12.862627 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:12.862910 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:12.862986 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:12.864572 (+ 1586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:19:12.864574 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:19:12.864574 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:19:12.864581 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:19:12.866311 (+ 1730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:19:12.866733 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:19:12.866735 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:19:12.866735 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:19:12.866794 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:19:12.867008 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:19:12.867008 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:19:12.867009 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:19:12.867134 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:19:12.867211 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0929 03:19:12.867456 (+ 245us) server_negotiation.cc:299] Negotiation successful
0929 03:19:12.867533 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260929 03:19:12.869984 32412 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32900:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 03:19:12.872692 32412 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 03:19:12.882853 443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:12.876113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.66:33431 (local address 127.0.0.1:36484)
0929 03:19:12.877136 (+ 1023us) negotiation.cc:107] Waiting for socket to connect
0929 03:19:12.877179 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 03:19:12.877244 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:19:12.877860 (+ 616us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:19:12.877864 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:19:12.877867 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:19:12.877941 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:19:12.877943 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:19:12.879632 (+ 1689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:19:12.879633 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:19:12.881454 (+ 1821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:19:12.881457 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:19:12.882532 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:19:12.882533 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:19:12.882628 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:12.882646 (+ 18us) client_negotiation.cc:769] Sending connection context
0929 03:19:12.882700 (+ 54us) client_negotiation.cc:240] Negotiation successful
0929 03:19:12.882752 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":991,"thread_start_us":70,"threads_started":1}
I20260929 03:19:12.883594 442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:12.876235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36484 (local address 127.20.178.66:33431)
0929 03:19:12.877679 (+ 1444us) server_negotiation.cc:206] Beginning negotiation
0929 03:19:12.877684 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:19:12.877695 (+ 11us) server_negotiation.cc:407] Connection header received
0929 03:19:12.877739 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:19:12.877741 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:19:12.877757 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:19:12.878010 (+ 253us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:19:12.878081 (+ 71us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:12.879524 (+ 1443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:12.881577 (+ 2053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:12.882416 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:12.883347 (+ 931us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:12.883446 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0929 03:19:12.883474 (+ 28us) server_negotiation.cc:299] Negotiation successful
0929 03:19:12.883515 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1398,"thread_start_us":88,"threads_started":1}
I20260929 03:19:12.884898 32683 tablet_service.cc:1467] Processing CreateTablet for tablet 50d990d48c6b44c4a5289fc397d1a1d9 (DEFAULT_TABLE table=test-table [id=e48b3e449dd7478d8cf35172cffd5920]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:19:12.885336 32683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50d990d48c6b44c4a5289fc397d1a1d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:19:12.890518 444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:12.876328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.67:43481 (local address 127.0.0.1:46752)
0929 03:19:12.877365 (+ 1037us) negotiation.cc:107] Waiting for socket to connect
0929 03:19:12.877374 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 03:19:12.877411 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:19:12.885002 (+ 7591us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:19:12.885003 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:19:12.885006 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:19:12.885048 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:19:12.885049 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:19:12.886407 (+ 1358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:19:12.886408 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:19:12.887537 (+ 1129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:19:12.887540 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:19:12.890287 (+ 2747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:19:12.890288 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:19:12.890367 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:12.890372 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 03:19:12.890402 (+ 30us) client_negotiation.cc:240] Negotiation successful
0929 03:19:12.890443 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1023,"thread_start_us":74,"threads_started":1}
I20260929 03:19:12.891106 446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:12.876811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46752 (local address 127.20.178.67:43481)
0929 03:19:12.884819 (+ 8008us) server_negotiation.cc:206] Beginning negotiation
0929 03:19:12.884825 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:19:12.884836 (+ 11us) server_negotiation.cc:407] Connection header received
0929 03:19:12.884877 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:19:12.884879 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:19:12.884896 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:19:12.885177 (+ 281us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:19:12.885253 (+ 76us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:12.886281 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:12.887899 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:12.888655 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:12.890873 (+ 2218us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:12.890965 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0929 03:19:12.890992 (+ 27us) server_negotiation.cc:299] Negotiation successful
0929 03:19:12.891030 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7967,"thread_start_us":74,"threads_started":1}
I20260929 03:19:12.891901 362 tablet_service.cc:1467] Processing CreateTablet for tablet 50d990d48c6b44c4a5289fc397d1a1d9 (DEFAULT_TABLE table=test-table [id=e48b3e449dd7478d8cf35172cffd5920]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:19:12.892238 362 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50d990d48c6b44c4a5289fc397d1a1d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:19:12.898419 441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:12.876551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.65:38029 (local address 127.0.0.1:47992)
0929 03:19:12.877489 (+ 938us) negotiation.cc:107] Waiting for socket to connect
0929 03:19:12.877497 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0929 03:19:12.877533 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:19:12.888952 (+ 11419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:19:12.888953 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:19:12.888956 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:19:12.888996 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:19:12.888998 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:19:12.896011 (+ 7013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:19:12.896012 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:19:12.897193 (+ 1181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:19:12.897196 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:19:12.898169 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:19:12.898170 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:19:12.898234 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:12.898239 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 03:19:12.898269 (+ 30us) client_negotiation.cc:240] Negotiation successful
0929 03:19:12.898329 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":919,"thread_start_us":63,"threads_started":1}
I20260929 03:19:12.898972 445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:12.876663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47992 (local address 127.20.178.65:38029)
0929 03:19:12.888787 (+ 12124us) server_negotiation.cc:206] Beginning negotiation
0929 03:19:12.888792 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:19:12.888802 (+ 10us) server_negotiation.cc:407] Connection header received
0929 03:19:12.888834 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:19:12.888836 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:19:12.888852 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:19:12.889064 (+ 212us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:19:12.889126 (+ 62us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:12.890173 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:12.897301 (+ 7128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:12.898067 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:12.898751 (+ 684us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:12.898827 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0929 03:19:12.898858 (+ 31us) server_negotiation.cc:299] Negotiation successful
0929 03:19:12.898899 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12087,"thread_start_us":55,"threads_started":1}
I20260929 03:19:12.899490 448 tablet_bootstrap.cc:492] T 50d990d48c6b44c4a5289fc397d1a1d9 P 369a835dde204fb182fc44de821a7f15: Bootstrap starting.
I20260929 03:19:12.899785 32556 tablet_service.cc:1467] Processing CreateTablet for tablet 50d990d48c6b44c4a5289fc397d1a1d9 (DEFAULT_TABLE table=test-table [id=e48b3e449dd7478d8cf35172cffd5920]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:19:12.900120 32556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50d990d48c6b44c4a5289fc397d1a1d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:19:12.900789 447 tablet_bootstrap.cc:492] T 50d990d48c6b44c4a5289fc397d1a1d9 P db3b3ccf47784156b2a0a3423ebec2d2: Bootstrap starting.
I20260929 03:19:12.901075 448 tablet_bootstrap.cc:654] T 50d990d48c6b44c4a5289fc397d1a1d9 P 369a835dde204fb182fc44de821a7f15: Neither blocks nor log segments found. Creating new log.
I20260929 03:19:12.901398 448 log.cc:824] T 50d990d48c6b44c4a5289fc397d1a1d9 P 369a835dde204fb182fc44de821a7f15: Log is configured to *not* fsync() on all Append() calls
I20260929 03:19:12.901939 447 tablet_bootstrap.cc:654] T 50d990d48c6b44c4a5289fc397d1a1d9 P db3b3ccf47784156b2a0a3423ebec2d2: Neither blocks nor log segments found. Creating new log.
I20260929 03:19:12.902181 448 tablet_bootstrap.cc:492] T 50d990d48c6b44c4a5289fc397d1a1d9 P 369a835dde204fb182fc44de821a7f15: No bootstrap required, opened a new log
I20260929 03:19:12.902257 448 ts_tablet_manager.cc:1397] T 50d990d48c6b44c4a5289fc397d1a1d9 P 369a835dde204fb182fc44de821a7f15: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260929 03:19:12.902426 447 log.cc:824] T 50d990d48c6b44c4a5289fc397d1a1d9 P db3b3ccf47784156b2a0a3423ebec2d2: Log is configured to *not* fsync() on all Append() calls
I20260929 03:19:12.904217 448 raft_consensus.cc:348] T 50d990d48c6b44c4a5289fc397d1a1d9 P 369a835dde204fb182fc44de821a7f15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db3b3ccf47784156b2a0a3423ebec2d2" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 33431 } } peers { permanent_uuid: "60fde20ef0914a439682ff4badecf345" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 38029 } } peers { permanent_uuid: "369a835dde204fb182fc44de821a7f15" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 43481 } }
I20260929 03:19:12.904398 448 raft_consensus.cc:374] T 50d990d48c6b44c4a5289fc397d1a1d9 P 369a835dde204fb182fc44de821a7f15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:19:12.904438 448 raft_consensus.cc:729] T 50d990d48c6b44c4a5289fc397d1a1d9 P 369a835dde204fb182fc44de821a7f15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 369a835dde204fb182fc44de821a7f15, State: Initialized, Role: FOLLOWER
I20260929 03:19:12.904552 448 consensus_queue.cc:260] T 50d990d48c6b44c4a5289fc397d1a1d9 P 369a835dde204fb182fc44de821a7f15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db3b3ccf47784156b2a0a3423ebec2d2" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 33431 } } peers { permanent_uuid: "60fde20ef0914a439682ff4badecf345" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 38029 } } peers { permanent_uuid: "369a835dde204fb182fc44de821a7f15" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 43481 } }
I20260929 03:19:12.904819 448 ts_tablet_manager.cc:1428] T 50d990d48c6b44c4a5289fc397d1a1d9 P 369a835dde204fb182fc44de821a7f15: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260929 03:19:12.905109 428 heartbeater.cc:503] Master 127.20.178.126:45575 was elected leader, sending a full tablet report...
I20260929 03:19:12.906710 451 tablet_bootstrap.cc:492] T 50d990d48c6b44c4a5289fc397d1a1d9 P 60fde20ef0914a439682ff4badecf345: Bootstrap starting.
I20260929 03:19:12.906951 447 tablet_bootstrap.cc:492] T 50d990d48c6b44c4a5289fc397d1a1d9 P db3b3ccf47784156b2a0a3423ebec2d2: No bootstrap required, opened a new log
I20260929 03:19:12.907011 447 ts_tablet_manager.cc:1397] T 50d990d48c6b44c4a5289fc397d1a1d9 P db3b3ccf47784156b2a0a3423ebec2d2: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260929 03:19:12.907682 451 tablet_bootstrap.cc:654] T 50d990d48c6b44c4a5289fc397d1a1d9 P 60fde20ef0914a439682ff4badecf345: Neither blocks nor log segments found. Creating new log.
I20260929 03:19:12.907938 451 log.cc:824] T 50d990d48c6b44c4a5289fc397d1a1d9 P 60fde20ef0914a439682ff4badecf345: Log is configured to *not* fsync() on all Append() calls
I20260929 03:19:12.908589 451 tablet_bootstrap.cc:492] T 50d990d48c6b44c4a5289fc397d1a1d9 P 60fde20ef0914a439682ff4badecf345: No bootstrap required, opened a new log
I20260929 03:19:12.908658 451 ts_tablet_manager.cc:1397] T 50d990d48c6b44c4a5289fc397d1a1d9 P 60fde20ef0914a439682ff4badecf345: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 03:19:12.909025 447 raft_consensus.cc:348] T 50d990d48c6b44c4a5289fc397d1a1d9 P db3b3ccf47784156b2a0a3423ebec2d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db3b3ccf47784156b2a0a3423ebec2d2" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 33431 } } peers { permanent_uuid: "60fde20ef0914a439682ff4badecf345" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 38029 } } peers { permanent_uuid: "369a835dde204fb182fc44de821a7f15" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 43481 } }
I20260929 03:19:12.909200 447 raft_consensus.cc:374] T 50d990d48c6b44c4a5289fc397d1a1d9 P db3b3ccf47784156b2a0a3423ebec2d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:19:12.909245 447 raft_consensus.cc:729] T 50d990d48c6b44c4a5289fc397d1a1d9 P db3b3ccf47784156b2a0a3423ebec2d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db3b3ccf47784156b2a0a3423ebec2d2, State: Initialized, Role: FOLLOWER
I20260929 03:19:12.909348 447 consensus_queue.cc:260] T 50d990d48c6b44c4a5289fc397d1a1d9 P db3b3ccf47784156b2a0a3423ebec2d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db3b3ccf47784156b2a0a3423ebec2d2" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 33431 } } peers { permanent_uuid: "60fde20ef0914a439682ff4badecf345" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 38029 } } peers { permanent_uuid: "369a835dde204fb182fc44de821a7f15" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 43481 } }
I20260929 03:19:12.909586 447 ts_tablet_manager.cc:1428] T 50d990d48c6b44c4a5289fc397d1a1d9 P db3b3ccf47784156b2a0a3423ebec2d2: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260929 03:19:12.909693 32762 heartbeater.cc:503] Master 127.20.178.126:45575 was elected leader, sending a full tablet report...
I20260929 03:19:12.910979 451 raft_consensus.cc:348] T 50d990d48c6b44c4a5289fc397d1a1d9 P 60fde20ef0914a439682ff4badecf345 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db3b3ccf47784156b2a0a3423ebec2d2" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 33431 } } peers { permanent_uuid: "60fde20ef0914a439682ff4badecf345" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 38029 } } peers { permanent_uuid: "369a835dde204fb182fc44de821a7f15" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 43481 } }
I20260929 03:19:12.911127 451 raft_consensus.cc:374] T 50d990d48c6b44c4a5289fc397d1a1d9 P 60fde20ef0914a439682ff4badecf345 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:19:12.911160 451 raft_consensus.cc:729] T 50d990d48c6b44c4a5289fc397d1a1d9 P 60fde20ef0914a439682ff4badecf345 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60fde20ef0914a439682ff4badecf345, State: Initialized, Role: FOLLOWER
I20260929 03:19:12.911252 451 consensus_queue.cc:260] T 50d990d48c6b44c4a5289fc397d1a1d9 P 60fde20ef0914a439682ff4badecf345 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db3b3ccf47784156b2a0a3423ebec2d2" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 33431 } } peers { permanent_uuid: "60fde20ef0914a439682ff4badecf345" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 38029 } } peers { permanent_uuid: "369a835dde204fb182fc44de821a7f15" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 43481 } }
I20260929 03:19:12.911486 451 ts_tablet_manager.cc:1428] T 50d990d48c6b44c4a5289fc397d1a1d9 P 60fde20ef0914a439682ff4badecf345: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260929 03:19:12.911613 32624 heartbeater.cc:503] Master 127.20.178.126:45575 was elected leader, sending a full tablet report...
Sep 29 03:19:12 dist-test-slave-c9qh krb5kdc[31936](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651952, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.126@KRBTEST.COM for HTTP/127.20.178.84@KRBTEST.COM
W20260929 03:19:13.060858 32763 tablet.cc:2369] T 50d990d48c6b44c4a5289fc397d1a1d9 P db3b3ccf47784156b2a0a3423ebec2d2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 03:19:13.065819 32625 tablet.cc:2369] T 50d990d48c6b44c4a5289fc397d1a1d9 P 60fde20ef0914a439682ff4badecf345: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 03:19:13.076483 429 tablet.cc:2369] T 50d990d48c6b44c4a5289fc397d1a1d9 P 369a835dde204fb182fc44de821a7f15: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 03:19:13.214933 452 raft_consensus.cc:482] T 50d990d48c6b44c4a5289fc397d1a1d9 P 369a835dde204fb182fc44de821a7f15 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:19:13.215099 452 raft_consensus.cc:504] T 50d990d48c6b44c4a5289fc397d1a1d9 P 369a835dde204fb182fc44de821a7f15 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db3b3ccf47784156b2a0a3423ebec2d2" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 33431 } } peers { permanent_uuid: "60fde20ef0914a439682ff4badecf345" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 38029 } } peers { permanent_uuid: "369a835dde204fb182fc44de821a7f15" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 43481 } }
I20260929 03:19:13.215513 452 leader_election.cc:288] T 50d990d48c6b44c4a5289fc397d1a1d9 P 369a835dde204fb182fc44de821a7f15 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db3b3ccf47784156b2a0a3423ebec2d2 (127.20.178.66:33431), 60fde20ef0914a439682ff4badecf345 (127.20.178.65:38029)
I20260929 03:19:13.219130 430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:13.215702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.66:33431 (local address 127.20.178.67:37191)
0929 03:19:13.215884 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0929 03:19:13.215895 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 03:19:13.215942 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:19:13.216147 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:19:13.216148 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:19:13.216149 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:19:13.216215 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:19:13.216216 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:19:13.217086 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:19:13.217087 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:19:13.218167 (+ 1080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:19:13.218170 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:19:13.218897 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:19:13.218898 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:19:13.218975 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:13.218978 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 03:19:13.219020 (+ 42us) client_negotiation.cc:240] Negotiation successful
0929 03:19:13.219066 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163}
I20260929 03:19:13.219373 442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:13.215821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.67:37191 (local address 127.20.178.66:33431)
0929 03:19:13.215869 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0929 03:19:13.215871 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:19:13.216020 (+ 149us) server_negotiation.cc:407] Connection header received
0929 03:19:13.216051 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:19:13.216052 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:19:13.216063 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:19:13.216104 (+ 41us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:19:13.216278 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:13.216998 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:13.218279 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:13.218795 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:13.218859 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:13.218867 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0929 03:19:13.219269 (+ 402us) server_negotiation.cc:299] Negotiation successful
0929 03:19:13.219304 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260929 03:19:13.219626 32714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50d990d48c6b44c4a5289fc397d1a1d9" candidate_uuid: "369a835dde204fb182fc44de821a7f15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db3b3ccf47784156b2a0a3423ebec2d2" is_pre_election: true
I20260929 03:19:13.219749 32714 raft_consensus.cc:2454] T 50d990d48c6b44c4a5289fc397d1a1d9 P db3b3ccf47784156b2a0a3423ebec2d2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 369a835dde204fb182fc44de821a7f15 in term 0.
I20260929 03:19:13.219941 316 leader_election.cc:302] T 50d990d48c6b44c4a5289fc397d1a1d9 P 369a835dde204fb182fc44de821a7f15 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 369a835dde204fb182fc44de821a7f15, db3b3ccf47784156b2a0a3423ebec2d2; no voters:
I20260929 03:19:13.220081 452 raft_consensus.cc:2781] T 50d990d48c6b44c4a5289fc397d1a1d9 P 369a835dde204fb182fc44de821a7f15 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 03:19:13.220126 452 raft_consensus.cc:482] T 50d990d48c6b44c4a5289fc397d1a1d9 P 369a835dde204fb182fc44de821a7f15 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 03:19:13.220152 452 raft_consensus.cc:3037] T 50d990d48c6b44c4a5289fc397d1a1d9 P 369a835dde204fb182fc44de821a7f15 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:19:13.220520 445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:13.216011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.67:53489 (local address 127.20.178.65:38029)
0929 03:19:13.216053 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0929 03:19:13.216055 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:19:13.216463 (+ 408us) server_negotiation.cc:407] Connection header received
0929 03:19:13.216571 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:19:13.216572 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:19:13.216582 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:19:13.216611 (+ 29us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:19:13.216863 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:13.217586 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:13.219009 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:13.219630 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:13.219680 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:13.219688 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0929 03:19:13.220408 (+ 720us) server_negotiation.cc:299] Negotiation successful
0929 03:19:13.220449 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260929 03:19:13.220907 456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:13.215708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.65:38029 (local address 127.20.178.67:53489)
0929 03:19:13.216392 (+ 684us) negotiation.cc:107] Waiting for socket to connect
0929 03:19:13.216403 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 03:19:13.216492 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:19:13.216710 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:19:13.216711 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:19:13.216713 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:19:13.216787 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:19:13.216789 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:19:13.217690 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:19:13.217690 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:19:13.218886 (+ 1196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:19:13.218889 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:19:13.220252 (+ 1363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:19:13.220253 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:19:13.220320 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:13.220323 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 03:19:13.220783 (+ 460us) client_negotiation.cc:240] Negotiation successful
0929 03:19:13.220828 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":664,"mutex_wait_us":7,"thread_start_us":78,"threads_started":1}
I20260929 03:19:13.221019 452 raft_consensus.cc:504] T 50d990d48c6b44c4a5289fc397d1a1d9 P 369a835dde204fb182fc44de821a7f15 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db3b3ccf47784156b2a0a3423ebec2d2" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 33431 } } peers { permanent_uuid: "60fde20ef0914a439682ff4badecf345" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 38029 } } peers { permanent_uuid: "369a835dde204fb182fc44de821a7f15" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 43481 } }
I20260929 03:19:13.221136 452 leader_election.cc:288] T 50d990d48c6b44c4a5289fc397d1a1d9 P 369a835dde204fb182fc44de821a7f15 [CANDIDATE]: Term 1 election: Requested vote from peers db3b3ccf47784156b2a0a3423ebec2d2 (127.20.178.66:33431), 60fde20ef0914a439682ff4badecf345 (127.20.178.65:38029)
I20260929 03:19:13.221385 32714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50d990d48c6b44c4a5289fc397d1a1d9" candidate_uuid: "369a835dde204fb182fc44de821a7f15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db3b3ccf47784156b2a0a3423ebec2d2"
I20260929 03:19:13.221469 32714 raft_consensus.cc:3037] T 50d990d48c6b44c4a5289fc397d1a1d9 P db3b3ccf47784156b2a0a3423ebec2d2 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:19:13.221505 32568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50d990d48c6b44c4a5289fc397d1a1d9" candidate_uuid: "369a835dde204fb182fc44de821a7f15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60fde20ef0914a439682ff4badecf345" is_pre_election: true
I20260929 03:19:13.221628 32568 raft_consensus.cc:2454] T 50d990d48c6b44c4a5289fc397d1a1d9 P 60fde20ef0914a439682ff4badecf345 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 369a835dde204fb182fc44de821a7f15 in term 0.
I20260929 03:19:13.221715 32578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50d990d48c6b44c4a5289fc397d1a1d9" candidate_uuid: "369a835dde204fb182fc44de821a7f15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60fde20ef0914a439682ff4badecf345"
I20260929 03:19:13.221757 32578 raft_consensus.cc:3037] T 50d990d48c6b44c4a5289fc397d1a1d9 P 60fde20ef0914a439682ff4badecf345 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:19:13.222307 32714 raft_consensus.cc:2454] T 50d990d48c6b44c4a5289fc397d1a1d9 P db3b3ccf47784156b2a0a3423ebec2d2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 369a835dde204fb182fc44de821a7f15 in term 1.
I20260929 03:19:13.222537 32578 raft_consensus.cc:2454] T 50d990d48c6b44c4a5289fc397d1a1d9 P 60fde20ef0914a439682ff4badecf345 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 369a835dde204fb182fc44de821a7f15 in term 1.
I20260929 03:19:13.222692 316 leader_election.cc:302] T 50d990d48c6b44c4a5289fc397d1a1d9 P 369a835dde204fb182fc44de821a7f15 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 369a835dde204fb182fc44de821a7f15, db3b3ccf47784156b2a0a3423ebec2d2; no voters:
I20260929 03:19:13.222785 452 raft_consensus.cc:2781] T 50d990d48c6b44c4a5289fc397d1a1d9 P 369a835dde204fb182fc44de821a7f15 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:19:13.222944 452 raft_consensus.cc:686] T 50d990d48c6b44c4a5289fc397d1a1d9 P 369a835dde204fb182fc44de821a7f15 [term 1 LEADER]: Becoming Leader. State: Replica: 369a835dde204fb182fc44de821a7f15, State: Running, Role: LEADER
I20260929 03:19:13.223029 452 consensus_queue.cc:237] T 50d990d48c6b44c4a5289fc397d1a1d9 P 369a835dde204fb182fc44de821a7f15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db3b3ccf47784156b2a0a3423ebec2d2" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 33431 } } peers { permanent_uuid: "60fde20ef0914a439682ff4badecf345" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 38029 } } peers { permanent_uuid: "369a835dde204fb182fc44de821a7f15" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 43481 } }
I20260929 03:19:13.223762 32412 catalog_manager.cc:5495] T 50d990d48c6b44c4a5289fc397d1a1d9 P 369a835dde204fb182fc44de821a7f15 reported cstate change: term changed from 0 to 1, leader changed from <none> to 369a835dde204fb182fc44de821a7f15 (127.20.178.67). New cstate: current_term: 1 leader_uuid: "369a835dde204fb182fc44de821a7f15" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db3b3ccf47784156b2a0a3423ebec2d2" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 33431 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "60fde20ef0914a439682ff4badecf345" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 38029 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "369a835dde204fb182fc44de821a7f15" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 43481 } health_report { overall_health: HEALTHY } } }
I20260929 03:19:13.235275 446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:13.232106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46766 (local address 127.20.178.67:43481)
0929 03:19:13.232282 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0929 03:19:13.232284 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:19:13.232290 (+ 6us) server_negotiation.cc:407] Connection header received
0929 03:19:13.232318 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:19:13.232319 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:19:13.232345 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:19:13.232408 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:19:13.232575 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:13.233285 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:13.234409 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:13.234682 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:13.234748 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:13.234948 (+ 200us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:19:13.235065 (+ 117us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:19:13.235141 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0929 03:19:13.235194 (+ 53us) server_negotiation.cc:299] Negotiation successful
0929 03:19:13.235212 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159}
I20260929 03:19:13.237133 32578 raft_consensus.cc:1261] T 50d990d48c6b44c4a5289fc397d1a1d9 P 60fde20ef0914a439682ff4badecf345 [term 1 FOLLOWER]: Refusing update from remote peer 369a835dde204fb182fc44de821a7f15: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:19:13.237151 32714 raft_consensus.cc:1261] T 50d990d48c6b44c4a5289fc397d1a1d9 P db3b3ccf47784156b2a0a3423ebec2d2 [term 1 FOLLOWER]: Refusing update from remote peer 369a835dde204fb182fc44de821a7f15: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:19:13.238534 452 consensus_queue.cc:1035] T 50d990d48c6b44c4a5289fc397d1a1d9 P 369a835dde204fb182fc44de821a7f15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db3b3ccf47784156b2a0a3423ebec2d2" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 33431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:19:13.240082 452 consensus_queue.cc:1035] T 50d990d48c6b44c4a5289fc397d1a1d9 P 369a835dde204fb182fc44de821a7f15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60fde20ef0914a439682ff4badecf345" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 38029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:19:13.242626 460 mvcc.cc:205] Tried to move back new op lower bound from 7334510400456429568 to 7334510400401879040. Current Snapshot: MvccSnapshot[applied={T|T < 7334510400456429568}]
I20260929 03:19:13.244166 463 mvcc.cc:205] Tried to move back new op lower bound from 7334510400456429568 to 7334510400401879040. Current Snapshot: MvccSnapshot[applied={T|T < 7334510400456429568 or (T in {7334510400456429568})}]
I20260929 03:19:13.245684 32412 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32900:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:19:13.245782 32412 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32900:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:19:13.246589 32412 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 503c3193be3949c7a4233f5f67be9568: Sending DeleteTablet for 3 replicas of tablet 50d990d48c6b44c4a5289fc397d1a1d9
I20260929 03:19:13.247092 32683 tablet_service.cc:1514] Processing DeleteTablet for tablet 50d990d48c6b44c4a5289fc397d1a1d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:19:13 UTC) from {username='kudu', principal='kudu/127.20.178.126@KRBTEST.COM'} at 127.0.0.1:36484
I20260929 03:19:13.247284 362 tablet_service.cc:1514] Processing DeleteTablet for tablet 50d990d48c6b44c4a5289fc397d1a1d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:19:13 UTC) from {username='kudu', principal='kudu/127.20.178.126@KRBTEST.COM'} at 127.0.0.1:46752
I20260929 03:19:13.247473 32556 tablet_service.cc:1514] Processing DeleteTablet for tablet 50d990d48c6b44c4a5289fc397d1a1d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:19:13 UTC) from {username='kudu', principal='kudu/127.20.178.126@KRBTEST.COM'} at 127.0.0.1:47992
I20260929 03:19:13.248365 21193 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuoaHbX/build/release/bin/kudu with pid 32493
I20260929 03:19:13.249817 461 mvcc.cc:205] Tried to move back new op lower bound from 7334510400456429568 to 7334510400401879040. Current Snapshot: MvccSnapshot[applied={T|T < 7334510400456429568}]
I20260929 03:19:13.252411 469 tablet_replica.cc:330] T 50d990d48c6b44c4a5289fc397d1a1d9 P 369a835dde204fb182fc44de821a7f15: stopping tablet replica
I20260929 03:19:13.253831 468 tablet_replica.cc:330] T 50d990d48c6b44c4a5289fc397d1a1d9 P db3b3ccf47784156b2a0a3423ebec2d2: stopping tablet replica
I20260929 03:19:13.253993 469 raft_consensus.cc:2229] T 50d990d48c6b44c4a5289fc397d1a1d9 P 369a835dde204fb182fc44de821a7f15 [term 1 LEADER]: Raft consensus shutting down.
I20260929 03:19:13.254150 468 raft_consensus.cc:2229] T 50d990d48c6b44c4a5289fc397d1a1d9 P db3b3ccf47784156b2a0a3423ebec2d2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:19:13.254303 469 raft_consensus.cc:2258] T 50d990d48c6b44c4a5289fc397d1a1d9 P 369a835dde204fb182fc44de821a7f15 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:19:13.254441 468 raft_consensus.cc:2258] T 50d990d48c6b44c4a5289fc397d1a1d9 P db3b3ccf47784156b2a0a3423ebec2d2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:19:13.256534 469 ts_tablet_manager.cc:1905] T 50d990d48c6b44c4a5289fc397d1a1d9 P 369a835dde204fb182fc44de821a7f15: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:19:13.259727 469 ts_tablet_manager.cc:1918] T 50d990d48c6b44c4a5289fc397d1a1d9 P 369a835dde204fb182fc44de821a7f15: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:19:13.259856 469 log.cc:1196] T 50d990d48c6b44c4a5289fc397d1a1d9 P 369a835dde204fb182fc44de821a7f15: Deleting WAL directory at /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651852476208-21193-0/minicluster-data/ts-2/wal/wals/50d990d48c6b44c4a5289fc397d1a1d9
I20260929 03:19:13.259931 468 ts_tablet_manager.cc:1905] T 50d990d48c6b44c4a5289fc397d1a1d9 P db3b3ccf47784156b2a0a3423ebec2d2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:19:13.260228 469 ts_tablet_manager.cc:1939] T 50d990d48c6b44c4a5289fc397d1a1d9 P 369a835dde204fb182fc44de821a7f15: Deleting consensus metadata
I20260929 03:19:13.260665 32397 catalog_manager.cc:4841] TS 369a835dde204fb182fc44de821a7f15 (127.20.178.67:43481): tablet 50d990d48c6b44c4a5289fc397d1a1d9 (table test-table [id=e48b3e449dd7478d8cf35172cffd5920]) successfully deleted
W20260929 03:19:13.261042 32396 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.20.178.65:38029 (error 108)
I20260929 03:19:13.261868 468 ts_tablet_manager.cc:1918] T 50d990d48c6b44c4a5289fc397d1a1d9 P db3b3ccf47784156b2a0a3423ebec2d2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:19:13.261916 441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:13.261784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.65:38029 (local address 127.0.0.1:47996)
0929 03:19:13.261832 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0929 03:19:13.261883 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.178.65:38029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
I20260929 03:19:13.261914 21193 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuoaHbX/build/release/bin/kudu with pid 32630
I20260929 03:19:13.261940 468 log.cc:1196] T 50d990d48c6b44c4a5289fc397d1a1d9 P db3b3ccf47784156b2a0a3423ebec2d2: Deleting WAL directory at /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651852476208-21193-0/minicluster-data/ts-1/wal/wals/50d990d48c6b44c4a5289fc397d1a1d9
W20260929 03:19:13.262871 32396 catalog_manager.cc:4584] TS 60fde20ef0914a439682ff4badecf345 (127.20.178.65:38029): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 50d990d48c6b44c4a5289fc397d1a1d9: Network error: Client connection negotiation failed: client connection to 127.20.178.65:38029: connect: Connection refused (error 111)
I20260929 03:19:13.268121 21193 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuoaHbX/build/release/bin/kudu with pid 32767
I20260929 03:19:13.268342 441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:13.268209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.66:33431 (local address 127.0.0.1:36486)
0929 03:19:13.268278 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0929 03:19:13.268303 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.178.66:33431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260929 03:19:13.274187 21193 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuoaHbX/build/release/bin/kudu with pid 32380
I20260929 03:19:13.279559 21193 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260929 03:19:13.613628 21193 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260929 03:19:13.613725 21193 mini_ranger.cc:66] Stopping Ranger...
I20260929 03:19:13.957894 21193 mini_ranger.cc:68] Stopped Ranger
2026-09-29 03:19:13.958 UTC [31966] LOG: received smart shutdown request
2026-09-29 03:19:13.961 UTC [31966] LOG: background worker "logical replication launcher" (PID 31975) exited with exit code 1
2026-09-29 03:19:13.962 UTC [31970] LOG: shutting down
2026-09-29 03:19:14.135 UTC [31966] LOG: database system is shut down
2026-09-29T03:19:14Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (45753 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651852476208-21193-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:19:14 dist-test-slave-c9qh krb5kdc[482](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:19:14 dist-test-slave-c9qh krb5kdc[482](info): set up 2 sockets
Sep 29 03:19:14 dist-test-slave-c9qh krb5kdc[482](info): commencing operation
krb5kdc: starting...
W20260929 03:19:15.280570 21193 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651852476208-21193-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651852476208-21193-0/krb5kdc/test-user.keytab.
Sep 29 03:19:15 dist-test-slave-c9qh krb5kdc[482](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651955, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:19:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:19:15Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.20.178.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.20.178.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.20.178.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651852476208-21193-0/krb5kdc/rangeradmin_127.20.178.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.20.178.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651852476208-21193-0/krb5kdc/rangeradmin_127.20.178.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.20.178.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.20.178.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.20.178.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651852476208-21193-0/krb5kdc/rangerlookup_127.20.178.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.20.178.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651852476208-21193-0/krb5kdc/rangerlookup_127.20.178.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.20.178.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.20.178.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.178.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651852476208-21193-0/krb5kdc/HTTP_127.20.178.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.20.178.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651852476208-21193-0/krb5kdc/HTTP_127.20.178.84@KRBTEST.COM.keytab.
I20260929 03:19:15.436901 21193 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651852476208-21193-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskuoaHbX/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651852476208-21193-0/minicluster-data/postgres -l logfile start
2026-09-29 03:19:17.915 UTC [512] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-29 03:19:17.915 UTC [512] LOG: listening on IPv4 address "127.20.178.84", port 54007
2026-09-29 03:19:17.919 UTC [512] LOG: listening on Unix socket "/tmp/.s.PGSQL.54007"
2026-09-29 03:19:17.953 UTC [515] LOG: database system was shut down at 2026-09-29 03:19:16 UTC
2026-09-29 03:19:17.957 UTC [512] LOG: database system is ready to accept connections
I20260929 03:19:18.896497 21193 mini_postgres.cc:96] Postgres bound to 54007
2026-09-29 03:19:18.902 UTC [523] FATAL: database "slave" does not exist
127.20.178.84:54007 - accepting connections
I20260929 03:19:18.902875 21193 mini_ranger.cc:161] Starting Ranger...
I20260929 03:19:18.917119 21193 mini_ranger.cc:84] Created miniranger Postgres user
I20260929 03:19:19.393101 21193 mini_ranger.cc:87] Created ranger Postgres database
I20260929 03:19:19.393186 21193 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651852476208-21193-0/minicluster-data/ranger-admin
2026-09-29 03:19:19,659 [I] DB FLAVOR :POSTGRES
2026-09-29 03:19:19,660 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 03:19:19,660 [I] Checking connection..
2026-09-29 03:19:19,660 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuoaHbX/build/release/bin/postgresql.jar:/tmp/dist-test-taskuoaHbX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.178.84:54007/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-29 03:19:19,919 [I] Checking connection passed.
2026-09-29 03:19:19,919 [I] --------- Verifying version history table ---------
2026-09-29 03:19:19,919 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuoaHbX/build/release/bin/postgresql.jar:/tmp/dist-test-taskuoaHbX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.178.84:54007/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 03:19:20,144 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 03:19:20,145 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuoaHbX/build/release/bin/postgresql.jar:/tmp/dist-test-taskuoaHbX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.178.84:54007/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 03:19:20,389 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 03:19:20,389 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-29 03:19:20,390 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuoaHbX/build/release/bin/postgresql.jar:/tmp/dist-test-taskuoaHbX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.178.84:54007/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskuoaHbX/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-29 03:19:20.628 UTC [587] WARNING: there is no transaction in progress
2026-09-29 03:19:20,634 [I] create_dbversion_catalog.sql file imported successfully
2026-09-29 03:19:20,634 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuoaHbX/build/release/bin/postgresql.jar:/tmp/dist-test-taskuoaHbX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.178.84:54007/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 03:19:20,877 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-29 03:19:20,877 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 03:19:20,877 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuoaHbX/build/release/bin/postgresql.jar:/tmp/dist-test-taskuoaHbX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.178.84:54007/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-29 03:19:21,101 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuoaHbX/build/release/bin/postgresql.jar:/tmp/dist-test-taskuoaHbX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.178.84:54007/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-29 03:19:21,338 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuoaHbX/build/release/bin/postgresql.jar:/tmp/dist-test-taskuoaHbX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.178.84:54007/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-c9qh.c.gcp-upstream.internal','N') ;"
2026-09-29 03:19:21,555 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuoaHbX/build/release/bin/postgresql.jar:/tmp/dist-test-taskuoaHbX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.178.84:54007/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 03:19:21,788 [I] Table x_portal_user does not exist in database ranger
2026-09-29 03:19:21,788 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuoaHbX/build/release/bin/postgresql.jar:/tmp/dist-test-taskuoaHbX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.178.84:54007/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 03:19:22,045 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-29 03:19:22,045 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuoaHbX/build/release/bin/postgresql.jar:/tmp/dist-test-taskuoaHbX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.178.84:54007/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 03:19:22,303 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuoaHbX/build/release/bin/postgresql.jar:/tmp/dist-test-taskuoaHbX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.178.84:54007/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 03:19:22,540 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-29 03:19:22,540 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuoaHbX/build/release/bin/postgresql.jar:/tmp/dist-test-taskuoaHbX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.178.84:54007/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskuoaHbX/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-29 03:19:23.487 UTC [695] WARNING: there is no transaction in progress
2026-09-29 03:19:23.503 UTC [695] WARNING: there is no transaction in progress
2026-09-29 03:19:23.519 UTC [695] WARNING: there is no transaction in progress
2026-09-29 03:19:23.534 UTC [695] WARNING: there is no transaction in progress
2026-09-29 03:19:23.550 UTC [695] WARNING: there is no transaction in progress
2026-09-29 03:19:23.569 UTC [695] WARNING: there is no transaction in progress
2026-09-29 03:19:23.621 UTC [695] WARNING: there is no transaction in progress
2026-09-29 03:19:23.627 UTC [695] WARNING: there is no transaction in progress
2026-09-29 03:19:23.640 UTC [695] WARNING: there is no transaction in progress
2026-09-29 03:19:23.650 UTC [695] WARNING: there is no transaction in progress
2026-09-29 03:19:23.660 UTC [695] WARNING: there is no transaction in progress
2026-09-29 03:19:23.673 UTC [695] WARNING: there is no transaction in progress
2026-09-29 03:19:23.682 UTC [695] WARNING: there is no transaction in progress
2026-09-29 03:19:23.688 UTC [695] WARNING: there is no transaction in progress
2026-09-29 03:19:23.694 UTC [695] WARNING: there is no transaction in progress
2026-09-29 03:19:23.701 UTC [695] WARNING: there is no transaction in progress
2026-09-29 03:19:24,338 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-29 03:19:24,338 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuoaHbX/build/release/bin/postgresql.jar:/tmp/dist-test-taskuoaHbX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.178.84:54007/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-29 03:19:24,556 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-29 03:19:24,557 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuoaHbX/build/release/bin/postgresql.jar:/tmp/dist-test-taskuoaHbX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.178.84:54007/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 03:19:24,789 [I] Table x_portal_user already exists in database 'ranger'
2026-09-29 03:19:24,789 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuoaHbX/build/release/bin/postgresql.jar:/tmp/dist-test-taskuoaHbX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.178.84:54007/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 03:19:25,045 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-29 03:19:25,046 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuoaHbX/build/release/bin/postgresql.jar:/tmp/dist-test-taskuoaHbX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.178.84:54007/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 03:19:25,301 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuoaHbX/build/release/bin/postgresql.jar:/tmp/dist-test-taskuoaHbX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.178.84:54007/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 03:19:25,554 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuoaHbX/build/release/bin/postgresql.jar:/tmp/dist-test-taskuoaHbX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.178.84:54007/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-c9qh.c.gcp-upstream.internal';"
2026-09-29 03:19:25,793 [I] CORE_DB_SCHEMA import status has been updated
2026-09-29 03:19:25,794 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuoaHbX/build/release/bin/postgresql.jar:/tmp/dist-test-taskuoaHbX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.178.84:54007/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-29 03:19:26,045 [I] DB_PATCHES have already been applied
I20260929 03:19:26.051038 21193 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651852476208-21193-0/minicluster-data/ranger-admin:/tmp/dist-test-taskuoaHbX/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskuoaHbX/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskuoaHbX/build/release/bin/postgresql.jar:/tmp/dist-test-taskuoaHbX/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260929 03:19:26.051120 21193 mini_ranger.cc:193] Using host: 127.20.178.84
I20260929 03:19:26.054059 21193 mini_ranger.cc:227] Ranger admin URL: http://127.20.178.84:47877
Sep 29, 2026 3:19:26 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 3:19:26 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 3:19:26 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651852476208-21193-0/minicluster-data/ranger-admin/ews/webapp
Sep 29, 2026 3:19:26 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651852476208-21193-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 29, 2026 3:19:26 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651852476208-21193-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 29, 2026 3:19:26 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651852476208-21193-0/minicluster-data/ranger-admin/ews/webapp].
Sep 29, 2026 3:19:26 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260929 03:19:27.172571 21193 mini_ranger.cc:160] Time spent starting Ranger: real 8.270s user 0.000s sys 0.008s
Sep 29, 2026 3:19:45 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 29, 2026 3:19:45 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.XKeyREST
Sep 29, 2026 3:19:45 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 29, 2026 3:19:45 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 29, 2026 3:19:45 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 29, 2026 3:19:45 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 29, 2026 3:19:45 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 29, 2026 3:19:45 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 29, 2026 3:19:45 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 29, 2026 3:19:45 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 29, 2026 3:19:45 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 29, 2026 3:19:45 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 29, 2026 3:19:45 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 29, 2026 3:19:45 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 29, 2026 3:19:45 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 29, 2026 3:19:45 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 29, 2026 3:19:45 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 29, 2026 3:19:46 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260929 03:19:46.986169 21193 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.20.178.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.20.178.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.20.178.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651852476208-21193-0/krb5kdc/rangerkms_127.20.178.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.20.178.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651852476208-21193-0/krb5kdc/rangerkms_127.20.178.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.20.178.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.20.178.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.178.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651852476208-21193-0/krb5kdc/HTTP_127.20.178.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.20.178.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651852476208-21193-0/krb5kdc/HTTP_127.20.178.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 29 03:19:47 dist-test-slave-c9qh krb5kdc[482](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651987, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260929 03:19:47.071142 21193 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260929 03:19:47.080801 21193 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260929 03:19:47.966288 21193 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260929 03:19:47.966373 21193 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651852476208-21193-0/minicluster-data/ranger-kms
I20260929 03:19:47.966390 21193 mini_ranger_kms.cc:227] Using postgres at 127.20.178.84:54007
2026-09-29 03:19:48,122 [I] DB FLAVOR :POSTGRES
2026-09-29 03:19:48,123 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 03:19:48,123 [I] Checking connection
2026-09-29 03:19:48,123 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuoaHbX/build/release/bin/postgresql.jar:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651852476208-21193-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.178.84:54007/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-29 03:19:48,348 [I] connection success
2026-09-29 03:19:48,349 [I] --------- Verifying Ranger DB tables ---------
2026-09-29 03:19:48,349 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-29 03:19:48,349 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuoaHbX/build/release/bin/postgresql.jar:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651852476208-21193-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.178.84:54007/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-29 03:19:48,576 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-29 03:19:48,577 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 03:19:48,577 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-29 03:19:48,577 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuoaHbX/build/release/bin/postgresql.jar:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651852476208-21193-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.178.84:54007/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651852476208-21193-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-29 03:19:48,857 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260929 03:19:49.289815 21193 mini_ranger_kms.cc:310] Created kms service
I20260929 03:19:49.521687 21193 mini_ranger_kms.cc:326] Created kudu user
I20260929 03:19:49.565274 21193 mini_ranger_kms.cc:343] Created rangerkms user
Sep 29 03:19:49 dist-test-slave-c9qh krb5kdc[482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651987, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.20.178.84@KRBTEST.COM
I20260929 03:19:49.978587 21193 mini_ranger_kms.cc:384] Added ranger policy
I20260929 03:19:49.978703 21193 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651852476208-21193-0/minicluster-data/ranger-kms:/tmp/dist-test-taskuoaHbX/build/release/bin/postgresql.jar:/tmp/dist-test-taskuoaHbX/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskuoaHbX/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260929 03:19:49.978724 21193 mini_ranger_kms.cc:240] Using host: 127.20.178.84
I20260929 03:19:49.982578 21193 mini_ranger_kms.cc:276] Ranger KMS PID: 875
I20260929 03:19:49.982676 21193 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.20.178.84:41965
Sep 29, 2026 3:19:51 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 3:19:51 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 3:19:51 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651852476208-21193-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 29, 2026 3:19:51 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651852476208-21193-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 29, 2026 3:19:51 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651852476208-21193-0/minicluster-data/ranger-kms/ews/webapp].
Sep 29, 2026 3:19:51 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260929 03:19:52.247712 21193 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.177s user 0.002s sys 0.138s
I20260929 03:19:52.247823 21193 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260929 03:19:52.247879 21193 mini_ranger_kms.cc:401] 127.20.178.84:41965/kms/v1/keys
Sep 29 03:19:52 dist-test-slave-c9qh krb5kdc[482](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790651992, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.20.178.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 29 03:19:52 dist-test-slave-c9qh krb5kdc[482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651992, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.20.178.84@KRBTEST.COM for HTTP/127.20.178.84@KRBTEST.COM
Sep 29 03:19:53 dist-test-slave-c9qh krb5kdc[482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651992, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.20.178.84@KRBTEST.COM for HTTP/127.20.178.84@KRBTEST.COM
Sep 29 03:19:55 dist-test-slave-c9qh krb5kdc[482](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651995, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.20.178.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.178.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.178.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.126.keytab.
Entry for principal kudu/127.20.178.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.126.keytab.
I20260929 03:19:55.269452 21193 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuoaHbX/build/release/bin/kudu
/tmp/dist-test-taskuoaHbX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651852476208-21193-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651852476208-21193-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651852476208-21193-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651852476208-21193-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.178.126:41901
--webserver_interface=127.20.178.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.126.keytab
--principal=kudu/127.20.178.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.178.84:40979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.178.126:41901
--ranger_config_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651852476208-21193-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.20.178.84:41965/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651852476208-21193-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651852476208-21193-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651852476208-21193-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:19:55.350795 926 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:19:55.350996 926 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:19:55.351017 926 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:19:55.352202 926 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:19:55.352244 926 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:19:55.352258 926 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:19:55.352272 926 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:19:55.352283 926 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:19:55.353873 926 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.178.84:40979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651852476208-21193-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651852476208-21193-0/minicluster-data/master-0/wal
--ranger_kms_url=127.20.178.84:41965/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.20.178.126:41901
--ranger_config_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651852476208-21193-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.126.keytab
--principal=kudu/127.20.178.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.178.126:41901
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651852476208-21193-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.178.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.926
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651852476208-21193-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:19:55.354130 926 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:19:55.354355 926 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 03:19:55.360411 926 server_base.cc:1034] running on GCE node
W20260929 03:19:55.360446 934 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:19:55.360598 932 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:19:55.360539 931 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:19:55.360970 926 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:19:55.361289 926 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:19:55.362442 926 hybrid_clock.cc:648] HybridClock initialized: now 1790651995362419 us; error 41 us; skew 500 ppm
Sep 29 03:19:55 dist-test-slave-c9qh krb5kdc[482](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651995, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:19:55.369563 926 init.cc:377] Logged in from keytab as kudu/127.20.178.126@KRBTEST.COM (short username kudu)
I20260929 03:19:55.370042 926 webserver.cc:460] Webserver started at http://127.20.178.126:44337/ using document root <none> and password file <none>
I20260929 03:19:55.370285 926 fs_manager.cc:359] Metadata directory not provided
I20260929 03:19:55.370324 926 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:19:55.370446 926 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 03:19:55 dist-test-slave-c9qh krb5kdc[482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651995, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.126@KRBTEST.COM for HTTP/127.20.178.84@KRBTEST.COM
I20260929 03:19:55.414705 926 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651852476208-21193-0/minicluster-data/master-0/data/instance:
uuid: "0b0216bae06646e0b783a6f723f68608"
format_stamp: "Formatted at 2026-09-29 03:19:55 on dist-test-slave-c9qh"
server_key: "ad1685abcdebcb22091777966f2f5213"
server_key_iv: "351bb17cfdb3fae2afd7ff18967b3073"
server_key_version: "kuduclusterkey@0"
I20260929 03:19:55.415122 926 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651852476208-21193-0/minicluster-data/master-0/wal/instance:
uuid: "0b0216bae06646e0b783a6f723f68608"
format_stamp: "Formatted at 2026-09-29 03:19:55 on dist-test-slave-c9qh"
server_key: "ad1685abcdebcb22091777966f2f5213"
server_key_iv: "351bb17cfdb3fae2afd7ff18967b3073"
server_key_version: "kuduclusterkey@0"
I20260929 03:19:55.416563 926 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260929 03:19:55.427011 941 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:19:55.427206 926 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 03:19:55.427301 926 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651852476208-21193-0/minicluster-data/master-0/data,/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651852476208-21193-0/minicluster-data/master-0/wal
uuid: "0b0216bae06646e0b783a6f723f68608"
format_stamp: "Formatted at 2026-09-29 03:19:55 on dist-test-slave-c9qh"
server_key: "ad1685abcdebcb22091777966f2f5213"
server_key_iv: "351bb17cfdb3fae2afd7ff18967b3073"
server_key_version: "kuduclusterkey@0"
I20260929 03:19:55.427383 926 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651852476208-21193-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651852476208-21193-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651852476208-21193-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:19:55.439538 926 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:19:55.441635 926 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:19:55.441798 926 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:19:55.446578 926 rpc_server.cc:307] RPC server started. Bound to: 127.20.178.126:41901
I20260929 03:19:55.446614 993 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.178.126:41901 every 8 connection(s)
I20260929 03:19:55.446951 926 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651852476208-21193-0/minicluster-data/master-0/data/info.pb
I20260929 03:19:55.447741 994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:19:55.450886 994 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b0216bae06646e0b783a6f723f68608: Bootstrap starting.
I20260929 03:19:55.451582 994 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0b0216bae06646e0b783a6f723f68608: Neither blocks nor log segments found. Creating new log.
I20260929 03:19:55.451905 994 log.cc:824] T 00000000000000000000000000000000 P 0b0216bae06646e0b783a6f723f68608: Log is configured to *not* fsync() on all Append() calls
I20260929 03:19:55.452684 994 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b0216bae06646e0b783a6f723f68608: No bootstrap required, opened a new log
I20260929 03:19:55.454944 21193 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuoaHbX/build/release/bin/kudu as pid 926
I20260929 03:19:55.455057 21193 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651852476208-21193-0/minicluster-data/master-0/wal/instance
I20260929 03:19:55.455229 21193 external_mini_cluster.cc:1428] Setting key 873caf81e7c1e108233d5dbc45057839
I20260929 03:19:55.455600 994 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0b0216bae06646e0b783a6f723f68608 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b0216bae06646e0b783a6f723f68608" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 41901 } }
I20260929 03:19:55.455890 994 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0b0216bae06646e0b783a6f723f68608 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:19:55.456014 994 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0b0216bae06646e0b783a6f723f68608 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b0216bae06646e0b783a6f723f68608, State: Initialized, Role: FOLLOWER
I20260929 03:19:55.456182 994 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0b0216bae06646e0b783a6f723f68608 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b0216bae06646e0b783a6f723f68608" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 41901 } }
I20260929 03:19:55.456276 994 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0b0216bae06646e0b783a6f723f68608 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:19:55.456308 994 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0b0216bae06646e0b783a6f723f68608 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:19:55.456377 994 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0b0216bae06646e0b783a6f723f68608 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:19:55.457095 994 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0b0216bae06646e0b783a6f723f68608 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b0216bae06646e0b783a6f723f68608" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 41901 } }
I20260929 03:19:55.457204 994 leader_election.cc:302] T 00000000000000000000000000000000 P 0b0216bae06646e0b783a6f723f68608 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0b0216bae06646e0b783a6f723f68608; no voters:
I20260929 03:19:55.457388 994 leader_election.cc:288] T 00000000000000000000000000000000 P 0b0216bae06646e0b783a6f723f68608 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:19:55.457436 999 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0b0216bae06646e0b783a6f723f68608 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:19:55.457655 999 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0b0216bae06646e0b783a6f723f68608 [term 1 LEADER]: Becoming Leader. State: Replica: 0b0216bae06646e0b783a6f723f68608, State: Running, Role: LEADER
I20260929 03:19:55.457706 994 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0b0216bae06646e0b783a6f723f68608 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:19:55.457801 999 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0b0216bae06646e0b783a6f723f68608 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b0216bae06646e0b783a6f723f68608" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 41901 } }
I20260929 03:19:55.458177 999 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b0216bae06646e0b783a6f723f68608 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0b0216bae06646e0b783a6f723f68608" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b0216bae06646e0b783a6f723f68608" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 41901 } } }
I20260929 03:19:55.458297 999 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b0216bae06646e0b783a6f723f68608 [sys.catalog]: This master's current role is: LEADER
I20260929 03:19:55.458441 999 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b0216bae06646e0b783a6f723f68608 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0b0216bae06646e0b783a6f723f68608. Latest consensus state: current_term: 1 leader_uuid: "0b0216bae06646e0b783a6f723f68608" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b0216bae06646e0b783a6f723f68608" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 41901 } } }
I20260929 03:19:55.458520 999 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b0216bae06646e0b783a6f723f68608 [sys.catalog]: This master's current role is: LEADER
I20260929 03:19:55.458665 1006 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:19:55.459180 1006 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:19:55.459471 994 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651852476208-21193-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
Sep 29 03:19:55 dist-test-slave-c9qh krb5kdc[482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651995, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.178.126@KRBTEST.COM
I20260929 03:19:55.465857 1006 catalog_manager.cc:1347] Generated new cluster ID: c81482bee0bb45cb91c44f12dca4de78
I20260929 03:19:55.465955 1006 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:19:55.469954 998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:55.455749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47820 (local address 127.20.178.126:41901)
0929 03:19:55.455939 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0929 03:19:55.455949 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 03:19:55.455984 (+ 35us) server_negotiation.cc:407] Connection header received
0929 03:19:55.456125 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:19:55.456133 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:19:55.456192 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:19:55.456351 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:19:55.457157 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:55.458636 (+ 1479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:55.459707 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:55.462796 (+ 3089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:55.464978 (+ 2182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:55.466227 (+ 1249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:19:55.466231 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:19:55.466232 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:19:55.466243 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:19:55.468157 (+ 1914us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:19:55.468532 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:19:55.468535 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:19:55.468536 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:19:55.468652 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:19:55.468836 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:19:55.468837 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:19:55.468837 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:19:55.469107 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:19:55.469209 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0929 03:19:55.469642 (+ 433us) server_negotiation.cc:299] Negotiation successful
0929 03:19:55.469761 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":84,"threads_started":1}
I20260929 03:19:55.494609 1006 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:19:55.495368 1006 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:19:55.511538 1006 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0b0216bae06646e0b783a6f723f68608: Generated new TSK 0
I20260929 03:19:55.511875 1006 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.178.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.178.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.178.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.65.keytab.
Entry for principal kudu/127.20.178.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.65.keytab.
I20260929 03:19:56.201082 21193 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuoaHbX/build/release/bin/kudu
/tmp/dist-test-taskuoaHbX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651852476208-21193-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651852476208-21193-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651852476208-21193-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651852476208-21193-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.20.178.84:41965/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.178.65:0
--local_ip_for_outbound_sockets=127.20.178.65
--webserver_interface=127.20.178.65
--webserver_port=0
--tserver_master_addrs=127.20.178.126:41901
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.65.keytab
--principal=kudu/127.20.178.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.178.84:40979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651852476208-21193-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651852476208-21193-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651852476208-21193-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:19:56.321146 1039 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:19:56.321420 1039 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:19:56.321463 1039 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:19:56.323118 1039 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:19:56.323179 1039 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:19:56.323218 1039 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.178.65
I20260929 03:19:56.325300 1039 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.178.84:40979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651852476208-21193-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651852476208-21193-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.20.178.84:41965/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.65.keytab
--principal=kudu/127.20.178.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.178.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651852476208-21193-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.178.65
--webserver_port=0
--tserver_master_addrs=127.20.178.126:41901
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1039
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.178.65
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651852476208-21193-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:19:56.325524 1039 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:19:56.325820 1039 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:19:56.332118 1047 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:19:56.332248 1045 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:19:56.332137 1044 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:19:56.333194 1039 server_base.cc:1034] running on GCE node
I20260929 03:19:56.333420 1039 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:19:56.333678 1039 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:19:56.334831 1039 hybrid_clock.cc:648] HybridClock initialized: now 1790651996334817 us; error 32 us; skew 500 ppm
Sep 29 03:19:56 dist-test-slave-c9qh krb5kdc[482](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651996, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:19:56.337502 1039 init.cc:377] Logged in from keytab as kudu/127.20.178.65@KRBTEST.COM (short username kudu)
I20260929 03:19:56.337922 1039 webserver.cc:460] Webserver started at http://127.20.178.65:35549/ using document root <none> and password file <none>
I20260929 03:19:56.338171 1039 fs_manager.cc:359] Metadata directory not provided
I20260929 03:19:56.338238 1039 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:19:56.338392 1039 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 03:19:56 dist-test-slave-c9qh krb5kdc[482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651996, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.65@KRBTEST.COM for HTTP/127.20.178.84@KRBTEST.COM
I20260929 03:19:56.356125 1039 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651852476208-21193-0/minicluster-data/ts-0/data/instance:
uuid: "8e9d29d06b3b4e9cb85ad2c42b860961"
format_stamp: "Formatted at 2026-09-29 03:19:56 on dist-test-slave-c9qh"
server_key: "b4d0f6d7df5aeb4e21ff6661e9961054"
server_key_iv: "5e6374e138f32b6e826fc072833a8861"
server_key_version: "kuduclusterkey@0"
I20260929 03:19:56.356594 1039 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651852476208-21193-0/minicluster-data/ts-0/wal/instance:
uuid: "8e9d29d06b3b4e9cb85ad2c42b860961"
format_stamp: "Formatted at 2026-09-29 03:19:56 on dist-test-slave-c9qh"
server_key: "b4d0f6d7df5aeb4e21ff6661e9961054"
server_key_iv: "5e6374e138f32b6e826fc072833a8861"
server_key_version: "kuduclusterkey@0"
I20260929 03:19:56.358608 1039 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260929 03:19:56.379084 1054 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:19:56.379323 1039 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 03:19:56.379420 1039 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651852476208-21193-0/minicluster-data/ts-0/data,/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651852476208-21193-0/minicluster-data/ts-0/wal
uuid: "8e9d29d06b3b4e9cb85ad2c42b860961"
format_stamp: "Formatted at 2026-09-29 03:19:56 on dist-test-slave-c9qh"
server_key: "b4d0f6d7df5aeb4e21ff6661e9961054"
server_key_iv: "5e6374e138f32b6e826fc072833a8861"
server_key_version: "kuduclusterkey@0"
I20260929 03:19:56.379525 1039 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651852476208-21193-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651852476208-21193-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651852476208-21193-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:19:56.413065 1039 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:19:56.415550 1039 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:19:56.415699 1039 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:19:56.415961 1039 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:19:56.416373 1039 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:19:56.416425 1039 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:19:56.416456 1039 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:19:56.416513 1039 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:19:56.423799 1039 rpc_server.cc:307] RPC server started. Bound to: 127.20.178.65:42631
I20260929 03:19:56.423880 1169 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.178.65:42631 every 8 connection(s)
I20260929 03:19:56.424261 1039 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651852476208-21193-0/minicluster-data/ts-0/data/info.pb
I20260929 03:19:56.428325 21193 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuoaHbX/build/release/bin/kudu as pid 1039
I20260929 03:19:56.428407 21193 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651852476208-21193-0/minicluster-data/ts-0/wal/instance
I20260929 03:19:56.428577 21193 external_mini_cluster.cc:1428] Setting key 9efadcfdf570c1640bd54c4bc3bc3a7e
Sep 29 03:19:56 dist-test-slave-c9qh krb5kdc[482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651996, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.65@KRBTEST.COM for kudu/127.20.178.126@KRBTEST.COM
I20260929 03:19:56.435132 1173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:56.425112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:41901 (local address 127.20.178.65:58337)
0929 03:19:56.425349 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0929 03:19:56.425386 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 03:19:56.425941 (+ 555us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:19:56.426214 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:19:56.426219 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:19:56.426414 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:19:56.426608 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:19:56.426612 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:19:56.427319 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:19:56.427320 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:19:56.428155 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:19:56.428158 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:19:56.428752 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:19:56.428753 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:19:56.429064 (+ 311us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:56.429748 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:19:56.429759 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:19:56.431458 (+ 1699us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:19:56.433414 (+ 1956us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:19:56.433421 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:19:56.433423 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:19:56.433727 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:19:56.433963 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:19:56.433964 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:19:56.433964 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:19:56.434101 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:19:56.434406 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:19:56.434411 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:19:56.434573 (+ 162us) client_negotiation.cc:769] Sending connection context
0929 03:19:56.434679 (+ 106us) client_negotiation.cc:240] Negotiation successful
0929 03:19:56.434839 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":69,"threads_started":1}
I20260929 03:19:56.435791 1172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:56.425105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.65:58337 (local address 127.20.178.126:41901)
0929 03:19:56.425243 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0929 03:19:56.425245 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:19:56.425472 (+ 227us) server_negotiation.cc:407] Connection header received
0929 03:19:56.426049 (+ 577us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:19:56.426050 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:19:56.426065 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:19:56.426143 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:19:56.426704 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:56.427222 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:56.428255 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:56.428574 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:56.428665 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:56.431566 (+ 2901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:19:56.431567 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:19:56.431568 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:19:56.431576 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:19:56.433310 (+ 1734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:19:56.433816 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:19:56.433818 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:19:56.433818 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:19:56.433894 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:19:56.434179 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:19:56.434180 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:19:56.434180 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:19:56.434321 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:19:56.434380 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0929 03:19:56.435550 (+ 1170us) server_negotiation.cc:299] Negotiation successful
0929 03:19:56.435645 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":78,"threads_started":1}
I20260929 03:19:56.436254 1170 heartbeater.cc:347] Connected to a master server at 127.20.178.126:41901
I20260929 03:19:56.436347 1170 heartbeater.cc:464] Registering TS with master...
I20260929 03:19:56.436532 1170 heartbeater.cc:511] Master 127.20.178.126:41901 requested a full tablet report, sending...
I20260929 03:19:56.437158 958 ts_manager.cc:194] Registered new tserver with Master: 8e9d29d06b3b4e9cb85ad2c42b860961 (127.20.178.65:42631)
I20260929 03:19:56.438153 958 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.178.65@KRBTEST.COM'} at 127.20.178.65:58337
WARNING: no policy specified for kudu/127.20.178.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.178.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.178.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.66.keytab.
Entry for principal kudu/127.20.178.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.66.keytab.
I20260929 03:19:56.460423 21193 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuoaHbX/build/release/bin/kudu
/tmp/dist-test-taskuoaHbX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651852476208-21193-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651852476208-21193-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651852476208-21193-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651852476208-21193-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.20.178.84:41965/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.178.66:0
--local_ip_for_outbound_sockets=127.20.178.66
--webserver_interface=127.20.178.66
--webserver_port=0
--tserver_master_addrs=127.20.178.126:41901
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.66.keytab
--principal=kudu/127.20.178.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.178.84:40979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651852476208-21193-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651852476208-21193-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651852476208-21193-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 03:19:56 dist-test-slave-c9qh krb5kdc[482](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790651996, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260929 03:19:56.591948 1176 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:19:56.592319 1176 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:19:56.592402 1176 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:19:56.594254 1176 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:19:56.594362 1176 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:19:56.594450 1176 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.178.66
I20260929 03:19:56.596669 1176 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.178.84:40979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651852476208-21193-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651852476208-21193-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.20.178.84:41965/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.66.keytab
--principal=kudu/127.20.178.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.178.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651852476208-21193-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.178.66
--webserver_port=0
--tserver_master_addrs=127.20.178.126:41901
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1176
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.178.66
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651852476208-21193-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:19:56.597123 1176 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:19:56.597409 1176 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:19:56.603734 1184 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:19:56.603734 1186 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:19:56.604486 1183 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:19:56.605659 1176 server_base.cc:1034] running on GCE node
I20260929 03:19:56.605875 1176 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:19:56.606130 1176 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:19:56.609402 1176 hybrid_clock.cc:648] HybridClock initialized: now 1790651996609382 us; error 35 us; skew 500 ppm
Sep 29 03:19:56 dist-test-slave-c9qh krb5kdc[482](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651996, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:19:56.612771 1176 init.cc:377] Logged in from keytab as kudu/127.20.178.66@KRBTEST.COM (short username kudu)
I20260929 03:19:56.613384 1176 webserver.cc:460] Webserver started at http://127.20.178.66:41409/ using document root <none> and password file <none>
I20260929 03:19:56.613672 1176 fs_manager.cc:359] Metadata directory not provided
I20260929 03:19:56.613739 1176 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:19:56.613881 1176 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 03:19:56 dist-test-slave-c9qh krb5kdc[482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651996, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.66@KRBTEST.COM for HTTP/127.20.178.84@KRBTEST.COM
I20260929 03:19:56.630944 1176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651852476208-21193-0/minicluster-data/ts-1/data/instance:
uuid: "2faecac12db84219997ea188d2a149ac"
format_stamp: "Formatted at 2026-09-29 03:19:56 on dist-test-slave-c9qh"
server_key: "14a6d5b24b01f654505898c00033987b"
server_key_iv: "edca9c9053c500957e98204e854f9369"
server_key_version: "kuduclusterkey@0"
I20260929 03:19:56.631403 1176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651852476208-21193-0/minicluster-data/ts-1/wal/instance:
uuid: "2faecac12db84219997ea188d2a149ac"
format_stamp: "Formatted at 2026-09-29 03:19:56 on dist-test-slave-c9qh"
server_key: "14a6d5b24b01f654505898c00033987b"
server_key_iv: "edca9c9053c500957e98204e854f9369"
server_key_version: "kuduclusterkey@0"
I20260929 03:19:56.633510 1176 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260929 03:19:56.652019 1193 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:19:56.652287 1176 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.000s sys 0.000s
I20260929 03:19:56.652384 1176 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651852476208-21193-0/minicluster-data/ts-1/data,/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651852476208-21193-0/minicluster-data/ts-1/wal
uuid: "2faecac12db84219997ea188d2a149ac"
format_stamp: "Formatted at 2026-09-29 03:19:56 on dist-test-slave-c9qh"
server_key: "14a6d5b24b01f654505898c00033987b"
server_key_iv: "edca9c9053c500957e98204e854f9369"
server_key_version: "kuduclusterkey@0"
I20260929 03:19:56.652478 1176 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651852476208-21193-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651852476208-21193-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651852476208-21193-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:19:56.677397 1176 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:19:56.679883 1176 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:19:56.680039 1176 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:19:56.680248 1176 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:19:56.680590 1176 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:19:56.680660 1176 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:19:56.680691 1176 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:19:56.680709 1176 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:19:56.687220 1176 rpc_server.cc:307] RPC server started. Bound to: 127.20.178.66:38319
I20260929 03:19:56.687644 1176 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651852476208-21193-0/minicluster-data/ts-1/data/info.pb
I20260929 03:19:56.688856 1307 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.178.66:38319 every 8 connection(s)
I20260929 03:19:56.692584 21193 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuoaHbX/build/release/bin/kudu as pid 1176
I20260929 03:19:56.692746 21193 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651852476208-21193-0/minicluster-data/ts-1/wal/instance
I20260929 03:19:56.693044 21193 external_mini_cluster.cc:1428] Setting key 3e8cff98612bdc7e7a72b2ea2a19b251
Sep 29 03:19:56 dist-test-slave-c9qh krb5kdc[482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651996, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.66@KRBTEST.COM for kudu/127.20.178.126@KRBTEST.COM
I20260929 03:19:56.699442 1172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:56.688785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.66:40275 (local address 127.20.178.126:41901)
0929 03:19:56.688839 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0929 03:19:56.688841 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:19:56.688905 (+ 64us) server_negotiation.cc:407] Connection header received
0929 03:19:56.689753 (+ 848us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:19:56.689755 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:19:56.689773 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:19:56.689810 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:19:56.690733 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:56.691235 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:56.692956 (+ 1721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:56.693303 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:56.693439 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:56.696133 (+ 2694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:19:56.696135 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:19:56.696135 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:19:56.696144 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:19:56.698024 (+ 1880us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:19:56.698450 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:19:56.698451 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:19:56.698451 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:19:56.698517 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:19:56.698757 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:19:56.698758 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:19:56.698759 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:19:56.698905 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:19:56.698993 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0929 03:19:56.699238 (+ 245us) server_negotiation.cc:299] Negotiation successful
0929 03:19:56.699338 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260929 03:19:56.699461 1310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:56.688500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:41901 (local address 127.20.178.66:40275)
0929 03:19:56.688738 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0929 03:19:56.688797 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0929 03:19:56.689612 (+ 815us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:19:56.690038 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:19:56.690042 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:19:56.690309 (+ 267us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:19:56.690615 (+ 306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:19:56.690622 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:19:56.691338 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:19:56.691339 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:19:56.692845 (+ 1506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:19:56.692849 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:19:56.693463 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:19:56.693464 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:19:56.693624 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:56.694327 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:19:56.694339 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:19:56.696021 (+ 1682us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:19:56.698134 (+ 2113us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:19:56.698140 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:19:56.698142 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:19:56.698360 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:19:56.698597 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:19:56.698597 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:19:56.698598 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:19:56.698693 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:19:56.698986 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:19:56.698990 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:19:56.699104 (+ 114us) client_negotiation.cc:769] Sending connection context
0929 03:19:56.699185 (+ 81us) client_negotiation.cc:240] Negotiation successful
0929 03:19:56.699298 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":94,"threads_started":1}
I20260929 03:19:56.700289 1308 heartbeater.cc:347] Connected to a master server at 127.20.178.126:41901
I20260929 03:19:56.700407 1308 heartbeater.cc:464] Registering TS with master...
I20260929 03:19:56.700671 1308 heartbeater.cc:511] Master 127.20.178.126:41901 requested a full tablet report, sending...
I20260929 03:19:56.701071 958 ts_manager.cc:194] Registered new tserver with Master: 2faecac12db84219997ea188d2a149ac (127.20.178.66:38319)
I20260929 03:19:56.701606 958 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.178.66@KRBTEST.COM'} at 127.20.178.66:40275
WARNING: no policy specified for kudu/127.20.178.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.178.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.178.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.67.keytab.
Entry for principal kudu/127.20.178.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.67.keytab.
I20260929 03:19:56.728003 21193 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuoaHbX/build/release/bin/kudu
/tmp/dist-test-taskuoaHbX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651852476208-21193-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651852476208-21193-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651852476208-21193-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651852476208-21193-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.20.178.84:41965/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.178.67:0
--local_ip_for_outbound_sockets=127.20.178.67
--webserver_interface=127.20.178.67
--webserver_port=0
--tserver_master_addrs=127.20.178.126:41901
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.67.keytab
--principal=kudu/127.20.178.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.178.84:40979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651852476208-21193-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651852476208-21193-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651852476208-21193-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:19:56.841207 1313 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:19:56.841488 1313 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:19:56.841521 1313 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:19:56.843127 1313 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:19:56.843186 1313 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:19:56.843222 1313 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.178.67
I20260929 03:19:56.845335 1313 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.178.84:40979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651852476208-21193-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651852476208-21193-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.20.178.84:41965/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.67.keytab
--principal=kudu/127.20.178.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.178.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651852476208-21193-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.178.67
--webserver_port=0
--tserver_master_addrs=127.20.178.126:41901
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1313
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.178.67
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651852476208-21193-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:19:56.845588 1313 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:19:56.845865 1313 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:19:56.851320 1321 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:19:56.851344 1318 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:19:56.851370 1319 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:19:56.851702 1313 server_base.cc:1034] running on GCE node
I20260929 03:19:56.851912 1313 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:19:56.852202 1313 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:19:56.853382 1313 hybrid_clock.cc:648] HybridClock initialized: now 1790651996853367 us; error 28 us; skew 500 ppm
Sep 29 03:19:56 dist-test-slave-c9qh krb5kdc[482](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651996, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:19:56.856413 1313 init.cc:377] Logged in from keytab as kudu/127.20.178.67@KRBTEST.COM (short username kudu)
I20260929 03:19:56.856915 1313 webserver.cc:460] Webserver started at http://127.20.178.67:45811/ using document root <none> and password file <none>
I20260929 03:19:56.857225 1313 fs_manager.cc:359] Metadata directory not provided
I20260929 03:19:56.857288 1313 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:19:56.857429 1313 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 03:19:56 dist-test-slave-c9qh krb5kdc[482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651996, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.67@KRBTEST.COM for HTTP/127.20.178.84@KRBTEST.COM
I20260929 03:19:56.878871 1313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651852476208-21193-0/minicluster-data/ts-2/data/instance:
uuid: "d160edf54d534c67adcf0ef07968eefd"
format_stamp: "Formatted at 2026-09-29 03:19:56 on dist-test-slave-c9qh"
server_key: "41c029147fe1258363a802508332f92a"
server_key_iv: "9017d560dd597adfc53d317ad07face3"
server_key_version: "kuduclusterkey@0"
I20260929 03:19:56.879354 1313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651852476208-21193-0/minicluster-data/ts-2/wal/instance:
uuid: "d160edf54d534c67adcf0ef07968eefd"
format_stamp: "Formatted at 2026-09-29 03:19:56 on dist-test-slave-c9qh"
server_key: "41c029147fe1258363a802508332f92a"
server_key_iv: "9017d560dd597adfc53d317ad07face3"
server_key_version: "kuduclusterkey@0"
I20260929 03:19:56.881253 1313 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.004s
I20260929 03:19:56.902180 1328 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:19:56.902391 1313 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.001s sys 0.000s
I20260929 03:19:56.902486 1313 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651852476208-21193-0/minicluster-data/ts-2/data,/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651852476208-21193-0/minicluster-data/ts-2/wal
uuid: "d160edf54d534c67adcf0ef07968eefd"
format_stamp: "Formatted at 2026-09-29 03:19:56 on dist-test-slave-c9qh"
server_key: "41c029147fe1258363a802508332f92a"
server_key_iv: "9017d560dd597adfc53d317ad07face3"
server_key_version: "kuduclusterkey@0"
I20260929 03:19:56.902593 1313 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651852476208-21193-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651852476208-21193-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651852476208-21193-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:19:56.927292 1313 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:19:56.929873 1313 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:19:56.930029 1313 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:19:56.930317 1313 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:19:56.930792 1313 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:19:56.930842 1313 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:19:56.930881 1313 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:19:56.930905 1313 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:19:56.938387 1313 rpc_server.cc:307] RPC server started. Bound to: 127.20.178.67:45663
I20260929 03:19:56.938781 1313 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651852476208-21193-0/minicluster-data/ts-2/data/info.pb
I20260929 03:19:56.940877 1441 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.178.67:45663 every 8 connection(s)
Sep 29 03:19:56 dist-test-slave-c9qh krb5kdc[482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651996, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.67@KRBTEST.COM for kudu/127.20.178.126@KRBTEST.COM
I20260929 03:19:56.948863 21193 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuoaHbX/build/release/bin/kudu as pid 1313
I20260929 03:19:56.948961 21193 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651852476208-21193-0/minicluster-data/ts-2/wal/instance
I20260929 03:19:56.949160 21193 external_mini_cluster.cc:1428] Setting key 6bea033e55cb0fa94982287aa918d300
I20260929 03:19:56.951470 1172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:56.941755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.67:59863 (local address 127.20.178.126:41901)
0929 03:19:56.941812 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0929 03:19:56.941814 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:19:56.941829 (+ 15us) server_negotiation.cc:407] Connection header received
0929 03:19:56.942321 (+ 492us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:19:56.942322 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:19:56.942335 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:19:56.942384 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:19:56.942937 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:56.943435 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:56.945053 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:56.945349 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:56.945396 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:56.947712 (+ 2316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:19:56.947714 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:19:56.947714 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:19:56.947723 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:19:56.949452 (+ 1729us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:19:56.950328 (+ 876us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:19:56.950329 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:19:56.950330 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:19:56.950389 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:19:56.950718 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:19:56.950721 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:19:56.950721 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:19:56.950862 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:19:56.950906 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0929 03:19:56.951285 (+ 379us) server_negotiation.cc:299] Negotiation successful
0929 03:19:56.951373 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260929 03:19:56.951972 1444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:56.941488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:41901 (local address 127.20.178.67:59863)
0929 03:19:56.941666 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0929 03:19:56.941709 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 03:19:56.942200 (+ 491us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:19:56.942454 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:19:56.942458 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:19:56.942652 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:19:56.942843 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:19:56.942847 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:19:56.943563 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:19:56.943564 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:19:56.944947 (+ 1383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:19:56.944950 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:19:56.945506 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:19:56.945507 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:19:56.945672 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:56.946273 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:19:56.946283 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:19:56.947613 (+ 1330us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:19:56.949749 (+ 2136us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:19:56.949756 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:19:56.949758 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:19:56.950253 (+ 495us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:19:56.950474 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:19:56.950475 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:19:56.950475 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:19:56.950624 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:19:56.950955 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:19:56.950962 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:19:56.951127 (+ 165us) client_negotiation.cc:769] Sending connection context
0929 03:19:56.951629 (+ 502us) client_negotiation.cc:240] Negotiation successful
0929 03:19:56.951791 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":98,"threads_started":1}
I20260929 03:19:56.952431 1442 heartbeater.cc:347] Connected to a master server at 127.20.178.126:41901
I20260929 03:19:56.952546 1442 heartbeater.cc:464] Registering TS with master...
I20260929 03:19:56.952755 1442 heartbeater.cc:511] Master 127.20.178.126:41901 requested a full tablet report, sending...
I20260929 03:19:56.953164 958 ts_manager.cc:194] Registered new tserver with Master: d160edf54d534c67adcf0ef07968eefd (127.20.178.67:45663)
I20260929 03:19:56.953567 958 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.178.67@KRBTEST.COM'} at 127.20.178.67:59863
I20260929 03:19:56.960940 21193 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 03:19:56.961568 21193 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuoaHbX/build/release/bin/kudu with pid 1039
I20260929 03:19:56.966845 21193 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuoaHbX/build/release/bin/kudu with pid 1176
I20260929 03:19:56.971578 21193 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuoaHbX/build/release/bin/kudu with pid 1313
I20260929 03:19:56.977164 21193 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuoaHbX/build/release/bin/kudu with pid 926
I20260929 03:19:56.985529 21193 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuoaHbX/build/release/bin/kudu
/tmp/dist-test-taskuoaHbX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651852476208-21193-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651852476208-21193-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651852476208-21193-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651852476208-21193-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.178.126:41901
--webserver_interface=127.20.178.126
--webserver_port=44337
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.126.keytab
--principal=kudu/127.20.178.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.178.84:40979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.178.126:41901
--ranger_config_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651852476208-21193-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.20.178.84:41965/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.20.178.84:41965/kms with env {KRB5CCNAME=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651852476208-21193-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651852476208-21193-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651852476208-21193-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:19:57.065410 1445 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:19:57.065608 1445 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:19:57.065629 1445 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:19:57.066764 1445 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:19:57.066807 1445 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:19:57.066821 1445 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:19:57.066859 1445 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:19:57.066888 1445 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:19:57.068444 1445 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.178.84:40979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651852476208-21193-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651852476208-21193-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.20.178.84:41965/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.20.178.126:41901
--ranger_config_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651852476208-21193-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.126.keytab
--principal=kudu/127.20.178.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.178.126:41901
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651852476208-21193-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.178.126
--webserver_port=44337
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1445
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651852476208-21193-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:19:57.068722 1445 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:19:57.068943 1445 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:19:57.073649 1451 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:19:57.073697 1450 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:19:57.073828 1445 server_base.cc:1034] running on GCE node
W20260929 03:19:57.073648 1453 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:19:57.074213 1445 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:19:57.074497 1445 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:19:57.075664 1445 hybrid_clock.cc:648] HybridClock initialized: now 1790651997075653 us; error 44 us; skew 500 ppm
Sep 29 03:19:57 dist-test-slave-c9qh krb5kdc[482](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651997, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:19:57.078104 1445 init.cc:377] Logged in from keytab as kudu/127.20.178.126@KRBTEST.COM (short username kudu)
I20260929 03:19:57.078527 1445 webserver.cc:460] Webserver started at http://127.20.178.126:44337/ using document root <none> and password file <none>
I20260929 03:19:57.078760 1445 fs_manager.cc:359] Metadata directory not provided
I20260929 03:19:57.078835 1445 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 03:19:57 dist-test-slave-c9qh krb5kdc[482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651997, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.126@KRBTEST.COM for HTTP/127.20.178.84@KRBTEST.COM
I20260929 03:19:57.138499 1445 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 03:19:57.139227 1461 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:19:57.139379 1445 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 03:19:57.139493 1445 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651852476208-21193-0/minicluster-data/master-0/data,/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651852476208-21193-0/minicluster-data/master-0/wal
uuid: "0b0216bae06646e0b783a6f723f68608"
format_stamp: "Formatted at 2026-09-29 03:19:55 on dist-test-slave-c9qh"
server_key: "ad1685abcdebcb22091777966f2f5213"
server_key_iv: "351bb17cfdb3fae2afd7ff18967b3073"
server_key_version: "kuduclusterkey@0"
I20260929 03:19:57.139839 1445 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651852476208-21193-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651852476208-21193-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651852476208-21193-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:19:57.155032 1445 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:19:57.156612 1445 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:19:57.156831 1445 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:19:57.163273 1445 rpc_server.cc:307] RPC server started. Bound to: 127.20.178.126:41901
I20260929 03:19:57.163343 1513 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.178.126:41901 every 8 connection(s)
I20260929 03:19:57.163832 1445 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651852476208-21193-0/minicluster-data/master-0/data/info.pb
I20260929 03:19:57.165688 1514 sys_catalog.cc:263] Verifying existing consensus state
I20260929 03:19:57.166672 1514 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b0216bae06646e0b783a6f723f68608: Bootstrap starting.
I20260929 03:19:57.170936 21193 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuoaHbX/build/release/bin/kudu as pid 1445
I20260929 03:19:57.171263 21193 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuoaHbX/build/release/bin/kudu
/tmp/dist-test-taskuoaHbX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651852476208-21193-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651852476208-21193-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651852476208-21193-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651852476208-21193-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.20.178.84:41965/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.178.65:42631
--local_ip_for_outbound_sockets=127.20.178.65
--tserver_master_addrs=127.20.178.126:41901
--webserver_port=35549
--webserver_interface=127.20.178.65
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.65.keytab
--principal=kudu/127.20.178.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.178.84:40979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.20.178.84:41965/kms with env {KRB5CCNAME=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651852476208-21193-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651852476208-21193-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651852476208-21193-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 03:19:57.175526 1514 log.cc:824] T 00000000000000000000000000000000 P 0b0216bae06646e0b783a6f723f68608: Log is configured to *not* fsync() on all Append() calls
I20260929 03:19:57.177237 1514 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b0216bae06646e0b783a6f723f68608: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 03:19:57.177523 1514 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b0216bae06646e0b783a6f723f68608: Bootstrap complete.
I20260929 03:19:57.180032 1514 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0b0216bae06646e0b783a6f723f68608 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b0216bae06646e0b783a6f723f68608" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 41901 } }
I20260929 03:19:57.180258 1514 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0b0216bae06646e0b783a6f723f68608 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b0216bae06646e0b783a6f723f68608, State: Initialized, Role: FOLLOWER
I20260929 03:19:57.180426 1514 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0b0216bae06646e0b783a6f723f68608 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b0216bae06646e0b783a6f723f68608" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 41901 } }
I20260929 03:19:57.180497 1514 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0b0216bae06646e0b783a6f723f68608 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:19:57.180547 1514 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0b0216bae06646e0b783a6f723f68608 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:19:57.180605 1514 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0b0216bae06646e0b783a6f723f68608 [term 1 FOLLOWER]: Advancing to term 2
I20260929 03:19:57.181591 1514 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0b0216bae06646e0b783a6f723f68608 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b0216bae06646e0b783a6f723f68608" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 41901 } }
I20260929 03:19:57.181679 1514 leader_election.cc:302] T 00000000000000000000000000000000 P 0b0216bae06646e0b783a6f723f68608 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0b0216bae06646e0b783a6f723f68608; no voters:
I20260929 03:19:57.181864 1514 leader_election.cc:288] T 00000000000000000000000000000000 P 0b0216bae06646e0b783a6f723f68608 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 03:19:57.181923 1518 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0b0216bae06646e0b783a6f723f68608 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 03:19:57.182060 1518 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0b0216bae06646e0b783a6f723f68608 [term 2 LEADER]: Becoming Leader. State: Replica: 0b0216bae06646e0b783a6f723f68608, State: Running, Role: LEADER
I20260929 03:19:57.182130 1514 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0b0216bae06646e0b783a6f723f68608 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:19:57.182152 1518 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0b0216bae06646e0b783a6f723f68608 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b0216bae06646e0b783a6f723f68608" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 41901 } }
I20260929 03:19:57.182395 1519 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b0216bae06646e0b783a6f723f68608 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "0b0216bae06646e0b783a6f723f68608" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b0216bae06646e0b783a6f723f68608" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 41901 } } }
I20260929 03:19:57.182475 1519 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b0216bae06646e0b783a6f723f68608 [sys.catalog]: This master's current role is: LEADER
I20260929 03:19:57.182394 1518 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b0216bae06646e0b783a6f723f68608 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0b0216bae06646e0b783a6f723f68608. Latest consensus state: current_term: 2 leader_uuid: "0b0216bae06646e0b783a6f723f68608" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b0216bae06646e0b783a6f723f68608" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 41901 } } }
I20260929 03:19:57.182659 1518 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b0216bae06646e0b783a6f723f68608 [sys.catalog]: This master's current role is: LEADER
I20260929 03:19:57.182752 1522 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:19:57.183257 1522 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:19:57.183379 1522 catalog_manager.cc:1259] Loaded cluster ID: c81482bee0bb45cb91c44f12dca4de78
I20260929 03:19:57.183413 1522 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:19:57.183645 1514 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651852476208-21193-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260929 03:19:57.189591 1522 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:19:57.189976 1522 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 0b0216bae06646e0b783a6f723f68608: Loaded TSK: 0
I20260929 03:19:57.190371 1522 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 03:19:57.288130 1516 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:19:57.288403 1516 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:19:57.288439 1516 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:19:57.290171 1516 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:19:57.290244 1516 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:19:57.290288 1516 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.178.65
I20260929 03:19:57.292544 1516 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.178.84:40979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651852476208-21193-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651852476208-21193-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.20.178.84:41965/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.65.keytab
--principal=kudu/127.20.178.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.178.65:42631
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651852476208-21193-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.178.65
--webserver_port=35549
--tserver_master_addrs=127.20.178.126:41901
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1516
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.178.65
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651852476208-21193-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:19:57.292893 1516 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:19:57.293159 1516 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 03:19:57.302340 1516 server_base.cc:1034] running on GCE node
W20260929 03:19:57.305598 1554 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:19:57.305788 1551 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:19:57.305588 1552 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:19:57.306147 1516 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:19:57.306444 1516 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:19:57.307629 1516 hybrid_clock.cc:648] HybridClock initialized: now 1790651997307600 us; error 44 us; skew 500 ppm
Sep 29 03:19:57 dist-test-slave-c9qh krb5kdc[482](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651997, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:19:57.310940 1516 init.cc:377] Logged in from keytab as kudu/127.20.178.65@KRBTEST.COM (short username kudu)
I20260929 03:19:57.311457 1516 webserver.cc:460] Webserver started at http://127.20.178.65:35549/ using document root <none> and password file <none>
I20260929 03:19:57.311746 1516 fs_manager.cc:359] Metadata directory not provided
I20260929 03:19:57.311823 1516 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 03:19:57 dist-test-slave-c9qh krb5kdc[482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651997, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.65@KRBTEST.COM for HTTP/127.20.178.84@KRBTEST.COM
I20260929 03:19:57.332008 1516 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 03:19:57.333052 1562 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:19:57.333221 1516 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260929 03:19:57.333297 1516 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651852476208-21193-0/minicluster-data/ts-0/data,/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651852476208-21193-0/minicluster-data/ts-0/wal
uuid: "8e9d29d06b3b4e9cb85ad2c42b860961"
format_stamp: "Formatted at 2026-09-29 03:19:56 on dist-test-slave-c9qh"
server_key: "b4d0f6d7df5aeb4e21ff6661e9961054"
server_key_iv: "5e6374e138f32b6e826fc072833a8861"
server_key_version: "kuduclusterkey@0"
I20260929 03:19:57.333750 1516 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651852476208-21193-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651852476208-21193-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651852476208-21193-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:19:57.344901 1516 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:19:57.347262 1516 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:19:57.347514 1516 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:19:57.347810 1516 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:19:57.348263 1516 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:19:57.348321 1516 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:19:57.348361 1516 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:19:57.348404 1516 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:19:57.355893 1516 rpc_server.cc:307] RPC server started. Bound to: 127.20.178.65:42631
I20260929 03:19:57.356426 1516 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651852476208-21193-0/minicluster-data/ts-0/data/info.pb
I20260929 03:19:57.358717 21193 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuoaHbX/build/release/bin/kudu as pid 1516
I20260929 03:19:57.359061 21193 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuoaHbX/build/release/bin/kudu
/tmp/dist-test-taskuoaHbX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651852476208-21193-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651852476208-21193-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651852476208-21193-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651852476208-21193-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.20.178.84:41965/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.178.66:38319
--local_ip_for_outbound_sockets=127.20.178.66
--tserver_master_addrs=127.20.178.126:41901
--webserver_port=41409
--webserver_interface=127.20.178.66
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.66.keytab
--principal=kudu/127.20.178.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.178.84:40979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.20.178.84:41965/kms with env {KRB5CCNAME=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651852476208-21193-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651852476208-21193-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651852476208-21193-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 03:19:57.359324 1675 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.178.65:42631 every 8 connection(s)
Sep 29 03:19:57 dist-test-slave-c9qh krb5kdc[482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651997, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.65@KRBTEST.COM for kudu/127.20.178.126@KRBTEST.COM
I20260929 03:19:57.375531 1679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:57.361161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:41901 (local address 127.20.178.65:58325)
0929 03:19:57.361688 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0929 03:19:57.361737 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 03:19:57.362404 (+ 667us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:19:57.363236 (+ 832us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:19:57.363240 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:19:57.363509 (+ 269us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:19:57.363833 (+ 324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:19:57.363840 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:19:57.365880 (+ 2040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:19:57.365882 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:19:57.367375 (+ 1493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:19:57.367378 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:19:57.368297 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:19:57.368299 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:19:57.368488 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:57.369240 (+ 752us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:19:57.369254 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:19:57.371167 (+ 1913us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:19:57.373572 (+ 2405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:19:57.373579 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:19:57.373582 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:19:57.373906 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:19:57.374229 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:19:57.374230 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:19:57.374230 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:19:57.374365 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:19:57.374833 (+ 468us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:19:57.374838 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:19:57.375005 (+ 167us) client_negotiation.cc:769] Sending connection context
0929 03:19:57.375123 (+ 118us) client_negotiation.cc:240] Negotiation successful
0929 03:19:57.375282 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":95,"threads_started":1}
I20260929 03:19:57.375537 1680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:57.361227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.65:58325 (local address 127.20.178.126:41901)
0929 03:19:57.362632 (+ 1405us) server_negotiation.cc:206] Beginning negotiation
0929 03:19:57.362642 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 03:19:57.362682 (+ 40us) server_negotiation.cc:407] Connection header received
0929 03:19:57.362836 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:19:57.362843 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:19:57.362920 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:19:57.363078 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:19:57.363999 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:57.365726 (+ 1727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:57.367508 (+ 1782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:57.368157 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:57.368283 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:57.371299 (+ 3016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:19:57.371303 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:19:57.371304 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:19:57.371315 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:19:57.373476 (+ 2161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:19:57.374006 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:19:57.374009 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:19:57.374010 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:19:57.374129 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:19:57.374447 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:19:57.374448 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:19:57.374448 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:19:57.374751 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:19:57.374841 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0929 03:19:57.375205 (+ 364us) server_negotiation.cc:299] Negotiation successful
0929 03:19:57.375329 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1343,"thread_start_us":90,"threads_started":1}
I20260929 03:19:57.377130 1676 heartbeater.cc:347] Connected to a master server at 127.20.178.126:41901
I20260929 03:19:57.377241 1676 heartbeater.cc:464] Registering TS with master...
I20260929 03:19:57.377434 1676 heartbeater.cc:511] Master 127.20.178.126:41901 requested a full tablet report, sending...
W20260929 03:19:57.378371 1676 heartbeater.cc:644] Failed to heartbeat to 127.20.178.126:41901 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260929 03:19:57.378453 1676 heartbeater.cc:464] Registering TS with master...
I20260929 03:19:57.378522 1676 heartbeater.cc:511] Master 127.20.178.126:41901 requested a full tablet report, sending...
I20260929 03:19:57.378872 1676 heartbeater.cc:464] Registering TS with master...
I20260929 03:19:57.378965 1676 heartbeater.cc:511] Master 127.20.178.126:41901 requested a full tablet report, sending...
W20260929 03:19:57.379284 1676 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 03:19:57.475912 1678 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:19:57.476207 1678 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:19:57.476249 1678 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:19:57.478168 1678 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:19:57.478229 1678 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:19:57.478273 1678 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.178.66
I20260929 03:19:57.480609 1678 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.178.84:40979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651852476208-21193-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651852476208-21193-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.20.178.84:41965/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.66.keytab
--principal=kudu/127.20.178.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.178.66:38319
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651852476208-21193-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.178.66
--webserver_port=41409
--tserver_master_addrs=127.20.178.126:41901
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1678
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.178.66
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651852476208-21193-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:19:57.480896 1678 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:19:57.481169 1678 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:19:57.490101 1688 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:19:57.490096 1685 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:19:57.490096 1686 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:19:57.490676 1678 server_base.cc:1034] running on GCE node
I20260929 03:19:57.490931 1678 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:19:57.491236 1678 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:19:57.492448 1678 hybrid_clock.cc:648] HybridClock initialized: now 1790651997492425 us; error 36 us; skew 500 ppm
Sep 29 03:19:57 dist-test-slave-c9qh krb5kdc[482](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651997, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:19:57.495332 1678 init.cc:377] Logged in from keytab as kudu/127.20.178.66@KRBTEST.COM (short username kudu)
I20260929 03:19:57.495816 1678 webserver.cc:460] Webserver started at http://127.20.178.66:41409/ using document root <none> and password file <none>
I20260929 03:19:57.496061 1678 fs_manager.cc:359] Metadata directory not provided
I20260929 03:19:57.496145 1678 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 03:19:57 dist-test-slave-c9qh krb5kdc[482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651997, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.66@KRBTEST.COM for HTTP/127.20.178.84@KRBTEST.COM
I20260929 03:19:57.520007 1678 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 03:19:57.521021 1696 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:19:57.521205 1678 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 03:19:57.521342 1678 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651852476208-21193-0/minicluster-data/ts-1/data,/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651852476208-21193-0/minicluster-data/ts-1/wal
uuid: "2faecac12db84219997ea188d2a149ac"
format_stamp: "Formatted at 2026-09-29 03:19:56 on dist-test-slave-c9qh"
server_key: "14a6d5b24b01f654505898c00033987b"
server_key_iv: "edca9c9053c500957e98204e854f9369"
server_key_version: "kuduclusterkey@0"
I20260929 03:19:57.521730 1678 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651852476208-21193-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651852476208-21193-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651852476208-21193-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:19:57.535347 1678 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:19:57.538038 1678 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:19:57.538205 1678 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:19:57.538477 1678 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:19:57.538838 1678 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:19:57.538892 1678 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:19:57.538923 1678 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:19:57.538947 1678 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:19:57.546241 1678 rpc_server.cc:307] RPC server started. Bound to: 127.20.178.66:38319
I20260929 03:19:57.546350 1809 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.178.66:38319 every 8 connection(s)
I20260929 03:19:57.546665 1678 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651852476208-21193-0/minicluster-data/ts-1/data/info.pb
Sep 29 03:19:57 dist-test-slave-c9qh krb5kdc[482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651997, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.66@KRBTEST.COM for kudu/127.20.178.126@KRBTEST.COM
I20260929 03:19:57.556723 21193 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuoaHbX/build/release/bin/kudu as pid 1678
I20260929 03:19:57.557054 21193 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuoaHbX/build/release/bin/kudu
/tmp/dist-test-taskuoaHbX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651852476208-21193-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651852476208-21193-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651852476208-21193-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651852476208-21193-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.20.178.84:41965/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.178.67:45663
--local_ip_for_outbound_sockets=127.20.178.67
--tserver_master_addrs=127.20.178.126:41901
--webserver_port=45811
--webserver_interface=127.20.178.67
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.67.keytab
--principal=kudu/127.20.178.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.178.84:40979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.20.178.84:41965/kms with env {KRB5CCNAME=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651852476208-21193-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651852476208-21193-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651852476208-21193-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 03:19:57.557570 1680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:57.547568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.66:41459 (local address 127.20.178.126:41901)
0929 03:19:57.547617 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0929 03:19:57.547619 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:19:57.547895 (+ 276us) server_negotiation.cc:407] Connection header received
0929 03:19:57.548629 (+ 734us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:19:57.548656 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:19:57.548671 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:19:57.548738 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:19:57.549521 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:57.550033 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:57.551639 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:57.551828 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:57.551893 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:57.554396 (+ 2503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:19:57.554397 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:19:57.554397 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:19:57.554404 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:19:57.555888 (+ 1484us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:19:57.556393 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:19:57.556394 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:19:57.556394 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:19:57.556443 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:19:57.556781 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:19:57.556783 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:19:57.556783 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:19:57.556931 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:19:57.557006 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0929 03:19:57.557383 (+ 377us) server_negotiation.cc:299] Negotiation successful
0929 03:19:57.557475 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260929 03:19:57.558135 1812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:57.547559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:41901 (local address 127.20.178.66:41459)
0929 03:19:57.547732 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0929 03:19:57.547781 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 03:19:57.548481 (+ 700us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:19:57.548826 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:19:57.548830 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:19:57.549084 (+ 254us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:19:57.549417 (+ 333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:19:57.549423 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:19:57.550174 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:19:57.550175 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:19:57.551539 (+ 1364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:19:57.551542 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:19:57.551945 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:19:57.551947 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:19:57.552097 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:57.552663 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:19:57.552673 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:19:57.554279 (+ 1606us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:19:57.555973 (+ 1694us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:19:57.555980 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:19:57.555982 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:19:57.556291 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:19:57.556525 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:19:57.556525 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:19:57.556526 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:19:57.556701 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:19:57.557015 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:19:57.557023 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:19:57.557187 (+ 164us) client_negotiation.cc:769] Sending connection context
0929 03:19:57.557722 (+ 535us) client_negotiation.cc:240] Negotiation successful
0929 03:19:57.557897 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":93,"threads_started":1}
I20260929 03:19:57.558995 1810 heartbeater.cc:347] Connected to a master server at 127.20.178.126:41901
I20260929 03:19:57.559111 1810 heartbeater.cc:464] Registering TS with master...
I20260929 03:19:57.559319 1810 heartbeater.cc:511] Master 127.20.178.126:41901 requested a full tablet report, sending...
W20260929 03:19:57.559999 1810 heartbeater.cc:644] Failed to heartbeat to 127.20.178.126:41901 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260929 03:19:57.560088 1810 heartbeater.cc:464] Registering TS with master...
I20260929 03:19:57.560163 1810 heartbeater.cc:511] Master 127.20.178.126:41901 requested a full tablet report, sending...
I20260929 03:19:57.560575 1810 heartbeater.cc:464] Registering TS with master...
I20260929 03:19:57.560675 1810 heartbeater.cc:511] Master 127.20.178.126:41901 requested a full tablet report, sending...
W20260929 03:19:57.561021 1810 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 03:19:57.654947 1813 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:19:57.655175 1813 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:19:57.655216 1813 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:19:57.656358 1813 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:19:57.656424 1813 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:19:57.656471 1813 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.178.67
I20260929 03:19:57.658023 1813 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.178.84:40979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651852476208-21193-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651852476208-21193-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.20.178.84:41965/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.67.keytab
--principal=kudu/127.20.178.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.178.67:45663
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651852476208-21193-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.178.67
--webserver_port=45811
--tserver_master_addrs=127.20.178.126:41901
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1813
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.178.67
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651852476208-21193-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:19:57.658284 1813 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:19:57.658509 1813 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:19:57.664129 1818 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:19:57.664541 1821 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:19:57.664796 1819 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:19:57.665452 1813 server_base.cc:1034] running on GCE node
I20260929 03:19:57.665668 1813 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:19:57.665942 1813 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:19:57.667131 1813 hybrid_clock.cc:648] HybridClock initialized: now 1790651997667098 us; error 47 us; skew 500 ppm
Sep 29 03:19:57 dist-test-slave-c9qh krb5kdc[482](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651997, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:19:57.670502 1813 init.cc:377] Logged in from keytab as kudu/127.20.178.67@KRBTEST.COM (short username kudu)
I20260929 03:19:57.671072 1813 webserver.cc:460] Webserver started at http://127.20.178.67:45811/ using document root <none> and password file <none>
I20260929 03:19:57.671389 1813 fs_manager.cc:359] Metadata directory not provided
I20260929 03:19:57.671465 1813 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 03:19:57 dist-test-slave-c9qh krb5kdc[482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651997, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.67@KRBTEST.COM for HTTP/127.20.178.84@KRBTEST.COM
I20260929 03:19:57.692225 1813 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20260929 03:19:57.693035 1830 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:19:57.693186 1813 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 03:19:57.693264 1813 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651852476208-21193-0/minicluster-data/ts-2/data,/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651852476208-21193-0/minicluster-data/ts-2/wal
uuid: "d160edf54d534c67adcf0ef07968eefd"
format_stamp: "Formatted at 2026-09-29 03:19:56 on dist-test-slave-c9qh"
server_key: "41c029147fe1258363a802508332f92a"
server_key_iv: "9017d560dd597adfc53d317ad07face3"
server_key_version: "kuduclusterkey@0"
I20260929 03:19:57.694355 1813 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651852476208-21193-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651852476208-21193-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651852476208-21193-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:19:57.711401 1813 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:19:57.713557 1813 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:19:57.713718 1813 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:19:57.713948 1813 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:19:57.714314 1813 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:19:57.714365 1813 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:19:57.714416 1813 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:19:57.714442 1813 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:19:57.721230 1813 rpc_server.cc:307] RPC server started. Bound to: 127.20.178.67:45663
I20260929 03:19:57.721312 1943 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.178.67:45663 every 8 connection(s)
I20260929 03:19:57.721630 1813 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651852476208-21193-0/minicluster-data/ts-2/data/info.pb
I20260929 03:19:57.726336 21193 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuoaHbX/build/release/bin/kudu as pid 1813
Sep 29 03:19:57 dist-test-slave-c9qh krb5kdc[482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651997, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.67@KRBTEST.COM for kudu/127.20.178.126@KRBTEST.COM
I20260929 03:19:57.734076 1680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:57.722691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.67:46727 (local address 127.20.178.126:41901)
0929 03:19:57.722737 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0929 03:19:57.722740 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:19:57.723777 (+ 1037us) server_negotiation.cc:407] Connection header received
0929 03:19:57.724529 (+ 752us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:19:57.724531 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:19:57.724544 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:19:57.724593 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:19:57.725493 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:57.726247 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:57.727720 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:57.727916 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:57.727998 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:57.730564 (+ 2566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:19:57.730566 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:19:57.730566 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:19:57.730574 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:19:57.732334 (+ 1760us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:19:57.732897 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:19:57.732898 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:19:57.732899 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:19:57.732965 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:19:57.733236 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:19:57.733236 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:19:57.733237 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:19:57.733411 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:19:57.733480 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0929 03:19:57.733903 (+ 423us) server_negotiation.cc:299] Negotiation successful
0929 03:19:57.733984 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260929 03:19:57.734301 1946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:57.722419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:41901 (local address 127.20.178.67:46727)
0929 03:19:57.723607 (+ 1188us) negotiation.cc:107] Waiting for socket to connect
0929 03:19:57.723675 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0929 03:19:57.724379 (+ 704us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:19:57.724718 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:19:57.724723 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:19:57.725036 (+ 313us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:19:57.725345 (+ 309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:19:57.725353 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:19:57.726362 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:19:57.726364 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:19:57.727635 (+ 1271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:19:57.727637 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:19:57.728025 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:19:57.728027 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:19:57.728154 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:57.728867 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:19:57.728885 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:19:57.730473 (+ 1588us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:19:57.732431 (+ 1958us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:19:57.732437 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:19:57.732440 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:19:57.732776 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:19:57.733043 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:19:57.733044 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:19:57.733044 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:19:57.733157 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:19:57.733500 (+ 343us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:19:57.733508 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:19:57.733699 (+ 191us) client_negotiation.cc:769] Sending connection context
0929 03:19:57.733830 (+ 131us) client_negotiation.cc:240] Negotiation successful
0929 03:19:57.734022 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1136,"thread_start_us":78,"threads_started":1}
I20260929 03:19:57.736745 1944 heartbeater.cc:347] Connected to a master server at 127.20.178.126:41901
I20260929 03:19:57.736869 1944 heartbeater.cc:464] Registering TS with master...
I20260929 03:19:57.737048 1944 heartbeater.cc:511] Master 127.20.178.126:41901 requested a full tablet report, sending...
W20260929 03:19:57.737691 1944 heartbeater.cc:644] Failed to heartbeat to 127.20.178.126:41901 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260929 03:19:57.737773 1944 heartbeater.cc:464] Registering TS with master...
I20260929 03:19:57.737851 1944 heartbeater.cc:511] Master 127.20.178.126:41901 requested a full tablet report, sending...
I20260929 03:19:57.738174 1944 heartbeater.cc:464] Registering TS with master...
I20260929 03:19:57.738299 1944 heartbeater.cc:511] Master 127.20.178.126:41901 requested a full tablet report, sending...
W20260929 03:19:57.738595 1944 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260929 03:19:57.738889 1948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:57.727128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47824 (local address 127.20.178.126:41901)
0929 03:19:57.727262 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0929 03:19:57.727264 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:19:57.727395 (+ 131us) server_negotiation.cc:407] Connection header received
0929 03:19:57.727442 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:19:57.727443 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:19:57.727456 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:19:57.727529 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:19:57.728347 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:57.729056 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:57.731130 (+ 2074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:57.731445 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:57.731525 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:57.733055 (+ 1530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:19:57.733057 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:19:57.733057 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:19:57.733066 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:19:57.735771 (+ 2705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:19:57.736204 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:19:57.736205 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:19:57.736205 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:19:57.736277 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:19:57.738269 (+ 1992us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:19:57.738271 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:19:57.738271 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:19:57.738426 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:19:57.738622 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0929 03:19:57.738666 (+ 44us) server_negotiation.cc:299] Negotiation successful
0929 03:19:57.738760 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":75,"threads_started":1}
Sep 29 03:19:58 dist-test-slave-c9qh krb5kdc[482](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790651998, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:19:58.379472 1676 heartbeater.cc:464] Registering TS with master...
I20260929 03:19:58.379629 1676 heartbeater.cc:511] Master 127.20.178.126:41901 requested a full tablet report, sending...
I20260929 03:19:58.380015 1478 ts_manager.cc:194] Registered new tserver with Master: 8e9d29d06b3b4e9cb85ad2c42b860961 (127.20.178.65:42631)
I20260929 03:19:58.380733 1478 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.178.65@KRBTEST.COM'} at 127.20.178.65:58325
I20260929 03:19:58.561196 1810 heartbeater.cc:464] Registering TS with master...
I20260929 03:19:58.561359 1810 heartbeater.cc:511] Master 127.20.178.126:41901 requested a full tablet report, sending...
I20260929 03:19:58.561647 1478 ts_manager.cc:194] Registered new tserver with Master: 2faecac12db84219997ea188d2a149ac (127.20.178.66:38319)
I20260929 03:19:58.562215 1478 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.178.66@KRBTEST.COM'} at 127.20.178.66:41459
Sep 29 03:19:58 dist-test-slave-c9qh krb5kdc[482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651998, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.126@KRBTEST.COM for HTTP/127.20.178.84@KRBTEST.COM
I20260929 03:19:58.738802 1944 heartbeater.cc:464] Registering TS with master...
I20260929 03:19:58.739017 1944 heartbeater.cc:511] Master 127.20.178.126:41901 requested a full tablet report, sending...
I20260929 03:19:58.739419 1478 ts_manager.cc:194] Registered new tserver with Master: d160edf54d534c67adcf0ef07968eefd (127.20.178.67:45663)
I20260929 03:19:58.739858 1478 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.178.67@KRBTEST.COM'} at 127.20.178.67:46727
I20260929 03:19:58.741322 21193 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 03:19:58.751148 1966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:58.742266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48130 (local address 127.20.178.126:41901)
0929 03:19:58.744872 (+ 2606us) server_negotiation.cc:206] Beginning negotiation
0929 03:19:58.744874 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:19:58.744880 (+ 6us) server_negotiation.cc:407] Connection header received
0929 03:19:58.744901 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:19:58.744901 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:19:58.744910 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:19:58.745014 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:19:58.745547 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:58.746076 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:58.746798 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:58.747017 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:58.747097 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:58.748321 (+ 1224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:19:58.748323 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:19:58.748323 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:19:58.748331 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:19:58.749914 (+ 1583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:19:58.750280 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:19:58.750281 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:19:58.750281 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:19:58.750342 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:19:58.750552 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:19:58.750552 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:19:58.750553 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:19:58.750664 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:19:58.750774 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0929 03:19:58.750878 (+ 104us) server_negotiation.cc:299] Negotiation successful
0929 03:19:58.750962 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2586,"thread_start_us":63,"threads_started":1}
I20260929 03:19:58.753201 1478 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48130:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 03:19:58.754911 1478 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 03:19:58.770972 1974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:58.766244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.66:38319 (local address 127.0.0.1:47446)
0929 03:19:58.766616 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0929 03:19:58.766626 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 03:19:58.766667 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:19:58.767102 (+ 435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:19:58.767102 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:19:58.767104 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:19:58.767156 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:19:58.767157 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:19:58.768369 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:19:58.768370 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:19:58.769641 (+ 1271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:19:58.769644 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:19:58.770633 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:19:58.770634 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:19:58.770727 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:58.770744 (+ 17us) client_negotiation.cc:769] Sending connection context
0929 03:19:58.770830 (+ 86us) client_negotiation.cc:240] Negotiation successful
0929 03:19:58.770888 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":358,"mutex_wait_us":80,"thread_start_us":69,"threads_started":1}
I20260929 03:19:58.771272 1976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:58.766441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47446 (local address 127.20.178.66:38319)
0929 03:19:58.766946 (+ 505us) server_negotiation.cc:206] Beginning negotiation
0929 03:19:58.766951 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:19:58.766960 (+ 9us) server_negotiation.cc:407] Connection header received
0929 03:19:58.766991 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:19:58.766994 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:19:58.767009 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:19:58.767222 (+ 213us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:19:58.767301 (+ 79us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:58.768289 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:58.769742 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:58.770527 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:58.770609 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:58.770676 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0929 03:19:58.771150 (+ 474us) server_negotiation.cc:299] Negotiation successful
0929 03:19:58.771195 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":471,"thread_start_us":69,"threads_started":1}
I20260929 03:19:58.772053 1744 tablet_service.cc:1467] Processing CreateTablet for tablet 6287ccf63f744e43bb158f3ba1b68fd1 (DEFAULT_TABLE table=test-table [id=7bfe058f28c8474b952ee499ddda1f4e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:19:58.772406 1744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6287ccf63f744e43bb158f3ba1b68fd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:19:58.776181 1971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:58.765995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.67:45663 (local address 127.0.0.1:58992)
0929 03:19:58.766442 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0929 03:19:58.766452 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 03:19:58.766492 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:19:58.766792 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:19:58.766795 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:19:58.766797 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:19:58.766876 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:19:58.766878 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:19:58.768262 (+ 1384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:19:58.768263 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:19:58.774044 (+ 5781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:19:58.774048 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:19:58.775984 (+ 1936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:19:58.775985 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:19:58.776042 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:58.776047 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 03:19:58.776074 (+ 27us) client_negotiation.cc:240] Negotiation successful
0929 03:19:58.776112 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":431,"mutex_wait_us":4591,"thread_start_us":107,"threads_started":1}
I20260929 03:19:58.776762 1973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:58.766191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58992 (local address 127.20.178.67:45663)
0929 03:19:58.766612 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0929 03:19:58.766617 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:19:58.766629 (+ 12us) server_negotiation.cc:407] Connection header received
0929 03:19:58.766666 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:19:58.766669 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:19:58.766686 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:19:58.766963 (+ 277us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:19:58.767052 (+ 89us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:58.768169 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:58.775174 (+ 7005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:58.775879 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:58.776516 (+ 637us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:58.776596 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0929 03:19:58.776623 (+ 27us) server_negotiation.cc:299] Negotiation successful
0929 03:19:58.776691 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":383,"thread_start_us":62,"threads_started":1}
I20260929 03:19:58.777094 1977 tablet_bootstrap.cc:492] T 6287ccf63f744e43bb158f3ba1b68fd1 P 2faecac12db84219997ea188d2a149ac: Bootstrap starting.
I20260929 03:19:58.777537 1878 tablet_service.cc:1467] Processing CreateTablet for tablet 6287ccf63f744e43bb158f3ba1b68fd1 (DEFAULT_TABLE table=test-table [id=7bfe058f28c8474b952ee499ddda1f4e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:19:58.777742 1972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:58.765995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.65:42631 (local address 127.0.0.1:53662)
0929 03:19:58.766233 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0929 03:19:58.766276 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 03:19:58.766337 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:19:58.766800 (+ 463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:19:58.766801 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:19:58.766804 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:19:58.766958 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:19:58.766961 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:19:58.768347 (+ 1386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:19:58.768348 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:19:58.775074 (+ 6726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:19:58.775076 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:19:58.777535 (+ 2459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:19:58.777536 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:19:58.777607 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:58.777611 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 03:19:58.777642 (+ 31us) client_negotiation.cc:240] Negotiation successful
0929 03:19:58.777676 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209,"mutex_wait_us":5636,"thread_start_us":106,"threads_started":1}
I20260929 03:19:58.777876 1878 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6287ccf63f744e43bb158f3ba1b68fd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:19:58.777978 1977 tablet_bootstrap.cc:654] T 6287ccf63f744e43bb158f3ba1b68fd1 P 2faecac12db84219997ea188d2a149ac: Neither blocks nor log segments found. Creating new log.
I20260929 03:19:58.778126 1975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:58.766346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53662 (local address 127.20.178.65:42631)
0929 03:19:58.766495 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0929 03:19:58.766500 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:19:58.766512 (+ 12us) server_negotiation.cc:407] Connection header received
0929 03:19:58.766552 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:19:58.766555 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:19:58.766573 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:19:58.767070 (+ 497us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:19:58.767162 (+ 92us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:58.768250 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:58.776450 (+ 8200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:58.777429 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:58.777905 (+ 476us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:58.777972 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0929 03:19:58.778011 (+ 39us) server_negotiation.cc:299] Negotiation successful
0929 03:19:58.778055 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":59,"threads_started":1}
I20260929 03:19:58.778421 1977 log.cc:824] T 6287ccf63f744e43bb158f3ba1b68fd1 P 2faecac12db84219997ea188d2a149ac: Log is configured to *not* fsync() on all Append() calls
I20260929 03:19:58.779048 1610 tablet_service.cc:1467] Processing CreateTablet for tablet 6287ccf63f744e43bb158f3ba1b68fd1 (DEFAULT_TABLE table=test-table [id=7bfe058f28c8474b952ee499ddda1f4e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:19:58.779270 1977 tablet_bootstrap.cc:492] T 6287ccf63f744e43bb158f3ba1b68fd1 P 2faecac12db84219997ea188d2a149ac: No bootstrap required, opened a new log
I20260929 03:19:58.779349 1610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6287ccf63f744e43bb158f3ba1b68fd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:19:58.779421 1977 ts_tablet_manager.cc:1397] T 6287ccf63f744e43bb158f3ba1b68fd1 P 2faecac12db84219997ea188d2a149ac: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 03:19:58.781692 1979 tablet_bootstrap.cc:492] T 6287ccf63f744e43bb158f3ba1b68fd1 P d160edf54d534c67adcf0ef07968eefd: Bootstrap starting.
I20260929 03:19:58.782297 1979 tablet_bootstrap.cc:654] T 6287ccf63f744e43bb158f3ba1b68fd1 P d160edf54d534c67adcf0ef07968eefd: Neither blocks nor log segments found. Creating new log.
I20260929 03:19:58.782774 1979 log.cc:824] T 6287ccf63f744e43bb158f3ba1b68fd1 P d160edf54d534c67adcf0ef07968eefd: Log is configured to *not* fsync() on all Append() calls
I20260929 03:19:58.783115 1980 tablet_bootstrap.cc:492] T 6287ccf63f744e43bb158f3ba1b68fd1 P 8e9d29d06b3b4e9cb85ad2c42b860961: Bootstrap starting.
I20260929 03:19:58.782675 1977 raft_consensus.cc:348] T 6287ccf63f744e43bb158f3ba1b68fd1 P 2faecac12db84219997ea188d2a149ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e9d29d06b3b4e9cb85ad2c42b860961" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 42631 } } peers { permanent_uuid: "d160edf54d534c67adcf0ef07968eefd" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 45663 } } peers { permanent_uuid: "2faecac12db84219997ea188d2a149ac" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 38319 } }
I20260929 03:19:58.784307 1977 raft_consensus.cc:374] T 6287ccf63f744e43bb158f3ba1b68fd1 P 2faecac12db84219997ea188d2a149ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:19:58.784480 1979 tablet_bootstrap.cc:492] T 6287ccf63f744e43bb158f3ba1b68fd1 P d160edf54d534c67adcf0ef07968eefd: No bootstrap required, opened a new log
I20260929 03:19:58.784457 1977 raft_consensus.cc:729] T 6287ccf63f744e43bb158f3ba1b68fd1 P 2faecac12db84219997ea188d2a149ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2faecac12db84219997ea188d2a149ac, State: Initialized, Role: FOLLOWER
I20260929 03:19:58.784566 1979 ts_tablet_manager.cc:1397] T 6287ccf63f744e43bb158f3ba1b68fd1 P d160edf54d534c67adcf0ef07968eefd: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260929 03:19:58.784796 1977 consensus_queue.cc:260] T 6287ccf63f744e43bb158f3ba1b68fd1 P 2faecac12db84219997ea188d2a149ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e9d29d06b3b4e9cb85ad2c42b860961" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 42631 } } peers { permanent_uuid: "d160edf54d534c67adcf0ef07968eefd" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 45663 } } peers { permanent_uuid: "2faecac12db84219997ea188d2a149ac" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 38319 } }
I20260929 03:19:58.787469 1977 ts_tablet_manager.cc:1428] T 6287ccf63f744e43bb158f3ba1b68fd1 P 2faecac12db84219997ea188d2a149ac: Time spent starting tablet: real 0.008s user 0.000s sys 0.003s
I20260929 03:19:58.787658 1810 heartbeater.cc:503] Master 127.20.178.126:41901 was elected leader, sending a full tablet report...
I20260929 03:19:58.790766 1979 raft_consensus.cc:348] T 6287ccf63f744e43bb158f3ba1b68fd1 P d160edf54d534c67adcf0ef07968eefd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e9d29d06b3b4e9cb85ad2c42b860961" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 42631 } } peers { permanent_uuid: "d160edf54d534c67adcf0ef07968eefd" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 45663 } } peers { permanent_uuid: "2faecac12db84219997ea188d2a149ac" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 38319 } }
I20260929 03:19:58.791018 1979 raft_consensus.cc:374] T 6287ccf63f744e43bb158f3ba1b68fd1 P d160edf54d534c67adcf0ef07968eefd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:19:58.791111 1979 raft_consensus.cc:729] T 6287ccf63f744e43bb158f3ba1b68fd1 P d160edf54d534c67adcf0ef07968eefd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d160edf54d534c67adcf0ef07968eefd, State: Initialized, Role: FOLLOWER
I20260929 03:19:58.791291 1979 consensus_queue.cc:260] T 6287ccf63f744e43bb158f3ba1b68fd1 P d160edf54d534c67adcf0ef07968eefd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e9d29d06b3b4e9cb85ad2c42b860961" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 42631 } } peers { permanent_uuid: "d160edf54d534c67adcf0ef07968eefd" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 45663 } } peers { permanent_uuid: "2faecac12db84219997ea188d2a149ac" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 38319 } }
I20260929 03:19:58.791597 1979 ts_tablet_manager.cc:1428] T 6287ccf63f744e43bb158f3ba1b68fd1 P d160edf54d534c67adcf0ef07968eefd: Time spent starting tablet: real 0.007s user 0.004s sys 0.000s
I20260929 03:19:58.791747 1944 heartbeater.cc:503] Master 127.20.178.126:41901 was elected leader, sending a full tablet report...
I20260929 03:19:58.792220 1980 tablet_bootstrap.cc:654] T 6287ccf63f744e43bb158f3ba1b68fd1 P 8e9d29d06b3b4e9cb85ad2c42b860961: Neither blocks nor log segments found. Creating new log.
I20260929 03:19:58.792622 1980 log.cc:824] T 6287ccf63f744e43bb158f3ba1b68fd1 P 8e9d29d06b3b4e9cb85ad2c42b860961: Log is configured to *not* fsync() on all Append() calls
I20260929 03:19:58.793785 1980 tablet_bootstrap.cc:492] T 6287ccf63f744e43bb158f3ba1b68fd1 P 8e9d29d06b3b4e9cb85ad2c42b860961: No bootstrap required, opened a new log
I20260929 03:19:58.793855 1980 ts_tablet_manager.cc:1397] T 6287ccf63f744e43bb158f3ba1b68fd1 P 8e9d29d06b3b4e9cb85ad2c42b860961: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.000s
I20260929 03:19:58.795912 1980 raft_consensus.cc:348] T 6287ccf63f744e43bb158f3ba1b68fd1 P 8e9d29d06b3b4e9cb85ad2c42b860961 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e9d29d06b3b4e9cb85ad2c42b860961" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 42631 } } peers { permanent_uuid: "d160edf54d534c67adcf0ef07968eefd" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 45663 } } peers { permanent_uuid: "2faecac12db84219997ea188d2a149ac" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 38319 } }
I20260929 03:19:58.796064 1980 raft_consensus.cc:374] T 6287ccf63f744e43bb158f3ba1b68fd1 P 8e9d29d06b3b4e9cb85ad2c42b860961 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:19:58.796106 1980 raft_consensus.cc:729] T 6287ccf63f744e43bb158f3ba1b68fd1 P 8e9d29d06b3b4e9cb85ad2c42b860961 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e9d29d06b3b4e9cb85ad2c42b860961, State: Initialized, Role: FOLLOWER
I20260929 03:19:58.796231 1980 consensus_queue.cc:260] T 6287ccf63f744e43bb158f3ba1b68fd1 P 8e9d29d06b3b4e9cb85ad2c42b860961 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e9d29d06b3b4e9cb85ad2c42b860961" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 42631 } } peers { permanent_uuid: "d160edf54d534c67adcf0ef07968eefd" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 45663 } } peers { permanent_uuid: "2faecac12db84219997ea188d2a149ac" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 38319 } }
I20260929 03:19:58.796537 1676 heartbeater.cc:503] Master 127.20.178.126:41901 was elected leader, sending a full tablet report...
I20260929 03:19:58.796841 1980 ts_tablet_manager.cc:1428] T 6287ccf63f744e43bb158f3ba1b68fd1 P 8e9d29d06b3b4e9cb85ad2c42b860961: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
W20260929 03:19:58.797858 1811 tablet.cc:2369] T 6287ccf63f744e43bb158f3ba1b68fd1 P 2faecac12db84219997ea188d2a149ac: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 03:19:58.808154 1985 raft_consensus.cc:482] T 6287ccf63f744e43bb158f3ba1b68fd1 P 8e9d29d06b3b4e9cb85ad2c42b860961 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:19:58.808306 1985 raft_consensus.cc:504] T 6287ccf63f744e43bb158f3ba1b68fd1 P 8e9d29d06b3b4e9cb85ad2c42b860961 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e9d29d06b3b4e9cb85ad2c42b860961" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 42631 } } peers { permanent_uuid: "d160edf54d534c67adcf0ef07968eefd" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 45663 } } peers { permanent_uuid: "2faecac12db84219997ea188d2a149ac" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 38319 } }
I20260929 03:19:58.809540 1985 leader_election.cc:288] T 6287ccf63f744e43bb158f3ba1b68fd1 P 8e9d29d06b3b4e9cb85ad2c42b860961 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d160edf54d534c67adcf0ef07968eefd (127.20.178.67:45663), 2faecac12db84219997ea188d2a149ac (127.20.178.66:38319)
I20260929 03:19:58.813534 1973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:58.809066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.65:35131 (local address 127.20.178.67:45663)
0929 03:19:58.809110 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0929 03:19:58.809113 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:19:58.810037 (+ 924us) server_negotiation.cc:407] Connection header received
0929 03:19:58.810148 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:19:58.810149 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:19:58.810159 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:19:58.810189 (+ 30us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:19:58.810428 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:58.811192 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:58.812746 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:58.813142 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:58.813233 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:58.813243 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0929 03:19:58.813430 (+ 187us) server_negotiation.cc:299] Negotiation successful
0929 03:19:58.813471 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260929 03:19:58.814003 1986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:58.808859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.67:45663 (local address 127.20.178.65:35131)
0929 03:19:58.809963 (+ 1104us) negotiation.cc:107] Waiting for socket to connect
0929 03:19:58.809976 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 03:19:58.810070 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:19:58.810264 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:19:58.810265 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:19:58.810267 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:19:58.810350 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:19:58.810352 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:19:58.811295 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:19:58.811296 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:19:58.812600 (+ 1304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:19:58.812603 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:19:58.813249 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:19:58.813250 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:19:58.813340 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:58.813344 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 03:19:58.813422 (+ 78us) client_negotiation.cc:240] Negotiation successful
0929 03:19:58.813463 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1082,"thread_start_us":77,"threads_started":1}
I20260929 03:19:58.814355 1898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6287ccf63f744e43bb158f3ba1b68fd1" candidate_uuid: "8e9d29d06b3b4e9cb85ad2c42b860961" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d160edf54d534c67adcf0ef07968eefd" is_pre_election: true
I20260929 03:19:58.814527 1898 raft_consensus.cc:2454] T 6287ccf63f744e43bb158f3ba1b68fd1 P d160edf54d534c67adcf0ef07968eefd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e9d29d06b3b4e9cb85ad2c42b860961 in term 0.
I20260929 03:19:58.814743 1564 leader_election.cc:302] T 6287ccf63f744e43bb158f3ba1b68fd1 P 8e9d29d06b3b4e9cb85ad2c42b860961 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e9d29d06b3b4e9cb85ad2c42b860961, d160edf54d534c67adcf0ef07968eefd; no voters:
I20260929 03:19:58.814939 1985 raft_consensus.cc:2781] T 6287ccf63f744e43bb158f3ba1b68fd1 P 8e9d29d06b3b4e9cb85ad2c42b860961 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 03:19:58.815068 1985 raft_consensus.cc:482] T 6287ccf63f744e43bb158f3ba1b68fd1 P 8e9d29d06b3b4e9cb85ad2c42b860961 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 03:19:58.815147 1985 raft_consensus.cc:3037] T 6287ccf63f744e43bb158f3ba1b68fd1 P 8e9d29d06b3b4e9cb85ad2c42b860961 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:19:58.816080 1985 raft_consensus.cc:504] T 6287ccf63f744e43bb158f3ba1b68fd1 P 8e9d29d06b3b4e9cb85ad2c42b860961 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e9d29d06b3b4e9cb85ad2c42b860961" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 42631 } } peers { permanent_uuid: "d160edf54d534c67adcf0ef07968eefd" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 45663 } } peers { permanent_uuid: "2faecac12db84219997ea188d2a149ac" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 38319 } }
I20260929 03:19:58.816355 1985 leader_election.cc:288] T 6287ccf63f744e43bb158f3ba1b68fd1 P 8e9d29d06b3b4e9cb85ad2c42b860961 [CANDIDATE]: Term 1 election: Requested vote from peers d160edf54d534c67adcf0ef07968eefd (127.20.178.67:45663), 2faecac12db84219997ea188d2a149ac (127.20.178.66:38319)
I20260929 03:19:58.816377 1898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6287ccf63f744e43bb158f3ba1b68fd1" candidate_uuid: "8e9d29d06b3b4e9cb85ad2c42b860961" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d160edf54d534c67adcf0ef07968eefd"
I20260929 03:19:58.816459 1898 raft_consensus.cc:3037] T 6287ccf63f744e43bb158f3ba1b68fd1 P d160edf54d534c67adcf0ef07968eefd [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:19:58.817286 1898 raft_consensus.cc:2454] T 6287ccf63f744e43bb158f3ba1b68fd1 P d160edf54d534c67adcf0ef07968eefd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e9d29d06b3b4e9cb85ad2c42b860961 in term 1.
I20260929 03:19:58.817441 1564 leader_election.cc:302] T 6287ccf63f744e43bb158f3ba1b68fd1 P 8e9d29d06b3b4e9cb85ad2c42b860961 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e9d29d06b3b4e9cb85ad2c42b860961, d160edf54d534c67adcf0ef07968eefd; no voters:
I20260929 03:19:58.817561 1985 raft_consensus.cc:2781] T 6287ccf63f744e43bb158f3ba1b68fd1 P 8e9d29d06b3b4e9cb85ad2c42b860961 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:19:58.817704 1985 raft_consensus.cc:686] T 6287ccf63f744e43bb158f3ba1b68fd1 P 8e9d29d06b3b4e9cb85ad2c42b860961 [term 1 LEADER]: Becoming Leader. State: Replica: 8e9d29d06b3b4e9cb85ad2c42b860961, State: Running, Role: LEADER
I20260929 03:19:58.817819 1985 consensus_queue.cc:237] T 6287ccf63f744e43bb158f3ba1b68fd1 P 8e9d29d06b3b4e9cb85ad2c42b860961 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e9d29d06b3b4e9cb85ad2c42b860961" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 42631 } } peers { permanent_uuid: "d160edf54d534c67adcf0ef07968eefd" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 45663 } } peers { permanent_uuid: "2faecac12db84219997ea188d2a149ac" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 38319 } }
I20260929 03:19:58.818835 1478 catalog_manager.cc:5495] T 6287ccf63f744e43bb158f3ba1b68fd1 P 8e9d29d06b3b4e9cb85ad2c42b860961 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8e9d29d06b3b4e9cb85ad2c42b860961 (127.20.178.65). New cstate: current_term: 1 leader_uuid: "8e9d29d06b3b4e9cb85ad2c42b860961" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e9d29d06b3b4e9cb85ad2c42b860961" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 42631 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d160edf54d534c67adcf0ef07968eefd" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 45663 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2faecac12db84219997ea188d2a149ac" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 38319 } health_report { overall_health: UNKNOWN } } }
I20260929 03:19:58.819519 1976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:58.809469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.65:33521 (local address 127.20.178.66:38319)
0929 03:19:58.809517 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0929 03:19:58.809519 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:19:58.814758 (+ 5239us) server_negotiation.cc:407] Connection header received
0929 03:19:58.814862 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:19:58.814864 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:19:58.814874 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:19:58.814903 (+ 29us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:19:58.815074 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:58.815744 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:58.818128 (+ 2384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:58.818669 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:58.818707 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:58.818714 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0929 03:19:58.819392 (+ 678us) server_negotiation.cc:299] Negotiation successful
0929 03:19:58.819441 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260929 03:19:58.819911 1986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:58.809261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.66:38319 (local address 127.20.178.65:33521)
0929 03:19:58.814703 (+ 5442us) negotiation.cc:107] Waiting for socket to connect
0929 03:19:58.814715 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 03:19:58.814784 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:19:58.814967 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:19:58.814968 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:19:58.814969 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:19:58.814996 (+ 27us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:19:58.814997 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:19:58.815842 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:19:58.815843 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:19:58.818029 (+ 2186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:19:58.818031 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:19:58.819246 (+ 1215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:19:58.819247 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:19:58.819304 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:58.819307 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 03:19:58.819810 (+ 503us) client_negotiation.cc:240] Negotiation successful
0929 03:19:58.819849 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5431,"thread_start_us":85,"threads_started":1}
I20260929 03:19:58.820538 1764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6287ccf63f744e43bb158f3ba1b68fd1" candidate_uuid: "8e9d29d06b3b4e9cb85ad2c42b860961" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2faecac12db84219997ea188d2a149ac" is_pre_election: true
I20260929 03:19:58.820717 1764 raft_consensus.cc:2454] T 6287ccf63f744e43bb158f3ba1b68fd1 P 2faecac12db84219997ea188d2a149ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e9d29d06b3b4e9cb85ad2c42b860961 in term 0.
I20260929 03:19:58.820866 1763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6287ccf63f744e43bb158f3ba1b68fd1" candidate_uuid: "8e9d29d06b3b4e9cb85ad2c42b860961" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2faecac12db84219997ea188d2a149ac"
I20260929 03:19:58.820943 1763 raft_consensus.cc:3037] T 6287ccf63f744e43bb158f3ba1b68fd1 P 2faecac12db84219997ea188d2a149ac [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:19:58.821833 1763 raft_consensus.cc:2454] T 6287ccf63f744e43bb158f3ba1b68fd1 P 2faecac12db84219997ea188d2a149ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e9d29d06b3b4e9cb85ad2c42b860961 in term 1.
I20260929 03:19:58.830271 1975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:58.827778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53678 (local address 127.20.178.65:42631)
0929 03:19:58.827817 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0929 03:19:58.827820 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:19:58.827870 (+ 50us) server_negotiation.cc:407] Connection header received
0929 03:19:58.827955 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:19:58.827955 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:19:58.827972 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:19:58.828062 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:19:58.828189 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:58.828709 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:58.829461 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:58.829672 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:58.829757 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:58.829932 (+ 175us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:19:58.830046 (+ 114us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:19:58.830142 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0929 03:19:58.830162 (+ 20us) server_negotiation.cc:299] Negotiation successful
0929 03:19:58.830183 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260929 03:19:58.831959 1763 raft_consensus.cc:1261] T 6287ccf63f744e43bb158f3ba1b68fd1 P 2faecac12db84219997ea188d2a149ac [term 1 FOLLOWER]: Refusing update from remote peer 8e9d29d06b3b4e9cb85ad2c42b860961: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:19:58.832043 1898 raft_consensus.cc:1261] T 6287ccf63f744e43bb158f3ba1b68fd1 P d160edf54d534c67adcf0ef07968eefd [term 1 FOLLOWER]: Refusing update from remote peer 8e9d29d06b3b4e9cb85ad2c42b860961: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:19:58.832372 1988 consensus_queue.cc:1035] T 6287ccf63f744e43bb158f3ba1b68fd1 P 8e9d29d06b3b4e9cb85ad2c42b860961 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2faecac12db84219997ea188d2a149ac" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 38319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:19:58.832597 1985 consensus_queue.cc:1035] T 6287ccf63f744e43bb158f3ba1b68fd1 P 8e9d29d06b3b4e9cb85ad2c42b860961 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d160edf54d534c67adcf0ef07968eefd" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 45663 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:19:58.836472 1991 mvcc.cc:205] Tried to move back new op lower bound from 7334510587213225984 to 7334510587158532096. Current Snapshot: MvccSnapshot[applied={T|T < 7334510587213225984 or (T in {7334510587213225984})}]
I20260929 03:19:58.837031 1993 mvcc.cc:205] Tried to move back new op lower bound from 7334510587213225984 to 7334510587158532096. Current Snapshot: MvccSnapshot[applied={T|T < 7334510587213225984}]
I20260929 03:19:58.837544 1992 mvcc.cc:205] Tried to move back new op lower bound from 7334510587213225984 to 7334510587158532096. Current Snapshot: MvccSnapshot[applied={T|T < 7334510587213225984}]
I20260929 03:19:58.837910 1477 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48130:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:19:58.838011 1477 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48130:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:19:58.838745 1477 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0b0216bae06646e0b783a6f723f68608: Sending DeleteTablet for 3 replicas of tablet 6287ccf63f744e43bb158f3ba1b68fd1
I20260929 03:19:58.839052 1878 tablet_service.cc:1514] Processing DeleteTablet for tablet 6287ccf63f744e43bb158f3ba1b68fd1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:19:58 UTC) from {username='kudu', principal='kudu/127.20.178.126@KRBTEST.COM'} at 127.0.0.1:58992
I20260929 03:19:58.839215 1610 tablet_service.cc:1514] Processing DeleteTablet for tablet 6287ccf63f744e43bb158f3ba1b68fd1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:19:58 UTC) from {username='kudu', principal='kudu/127.20.178.126@KRBTEST.COM'} at 127.0.0.1:53662
I20260929 03:19:58.839407 2002 tablet_replica.cc:330] T 6287ccf63f744e43bb158f3ba1b68fd1 P 8e9d29d06b3b4e9cb85ad2c42b860961: stopping tablet replica
I20260929 03:19:58.839421 1744 tablet_service.cc:1514] Processing DeleteTablet for tablet 6287ccf63f744e43bb158f3ba1b68fd1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:19:58 UTC) from {username='kudu', principal='kudu/127.20.178.126@KRBTEST.COM'} at 127.0.0.1:47446
I20260929 03:19:58.839589 2002 raft_consensus.cc:2229] T 6287ccf63f744e43bb158f3ba1b68fd1 P 8e9d29d06b3b4e9cb85ad2c42b860961 [term 1 LEADER]: Raft consensus shutting down.
I20260929 03:19:58.839749 2002 raft_consensus.cc:2258] T 6287ccf63f744e43bb158f3ba1b68fd1 P 8e9d29d06b3b4e9cb85ad2c42b860961 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:19:58.839876 21193 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuoaHbX/build/release/bin/kudu with pid 1516
I20260929 03:19:58.841156 2003 tablet_replica.cc:330] T 6287ccf63f744e43bb158f3ba1b68fd1 P 2faecac12db84219997ea188d2a149ac: stopping tablet replica
I20260929 03:19:58.841252 2003 raft_consensus.cc:2229] T 6287ccf63f744e43bb158f3ba1b68fd1 P 2faecac12db84219997ea188d2a149ac [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:19:58.841356 2003 raft_consensus.cc:2258] T 6287ccf63f744e43bb158f3ba1b68fd1 P 2faecac12db84219997ea188d2a149ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:19:58.841786 2003 ts_tablet_manager.cc:1905] T 6287ccf63f744e43bb158f3ba1b68fd1 P 2faecac12db84219997ea188d2a149ac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:19:58.843827 2001 tablet_replica.cc:330] T 6287ccf63f744e43bb158f3ba1b68fd1 P d160edf54d534c67adcf0ef07968eefd: stopping tablet replica
I20260929 03:19:58.843943 2001 raft_consensus.cc:2229] T 6287ccf63f744e43bb158f3ba1b68fd1 P d160edf54d534c67adcf0ef07968eefd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:19:58.844077 2001 raft_consensus.cc:2258] T 6287ccf63f744e43bb158f3ba1b68fd1 P d160edf54d534c67adcf0ef07968eefd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:19:58.844239 2003 ts_tablet_manager.cc:1918] T 6287ccf63f744e43bb158f3ba1b68fd1 P 2faecac12db84219997ea188d2a149ac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:19:58.844309 2003 log.cc:1196] T 6287ccf63f744e43bb158f3ba1b68fd1 P 2faecac12db84219997ea188d2a149ac: Deleting WAL directory at /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651852476208-21193-0/minicluster-data/ts-1/wal/wals/6287ccf63f744e43bb158f3ba1b68fd1
I20260929 03:19:58.844579 2003 ts_tablet_manager.cc:1939] T 6287ccf63f744e43bb158f3ba1b68fd1 P 2faecac12db84219997ea188d2a149ac: Deleting consensus metadata
I20260929 03:19:58.845007 1465 catalog_manager.cc:4841] TS 2faecac12db84219997ea188d2a149ac (127.20.178.66:38319): tablet 6287ccf63f744e43bb158f3ba1b68fd1 (table test-table [id=7bfe058f28c8474b952ee499ddda1f4e]) successfully deleted
I20260929 03:19:58.846022 2001 ts_tablet_manager.cc:1905] T 6287ccf63f744e43bb158f3ba1b68fd1 P d160edf54d534c67adcf0ef07968eefd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:19:58.847839 2001 ts_tablet_manager.cc:1918] T 6287ccf63f744e43bb158f3ba1b68fd1 P d160edf54d534c67adcf0ef07968eefd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:19:58.847908 2001 log.cc:1196] T 6287ccf63f744e43bb158f3ba1b68fd1 P d160edf54d534c67adcf0ef07968eefd: Deleting WAL directory at /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651852476208-21193-0/minicluster-data/ts-2/wal/wals/6287ccf63f744e43bb158f3ba1b68fd1
W20260929 03:19:58.848100 1465 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.20.178.65:42631 (error 108)
I20260929 03:19:58.848196 2001 ts_tablet_manager.cc:1939] T 6287ccf63f744e43bb158f3ba1b68fd1 P d160edf54d534c67adcf0ef07968eefd: Deleting consensus metadata
I20260929 03:19:58.848695 1463 catalog_manager.cc:4841] TS d160edf54d534c67adcf0ef07968eefd (127.20.178.67:45663): tablet 6287ccf63f744e43bb158f3ba1b68fd1 (table test-table [id=7bfe058f28c8474b952ee499ddda1f4e]) successfully deleted
I20260929 03:19:58.848842 21193 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuoaHbX/build/release/bin/kudu with pid 1678
I20260929 03:19:58.849376 1972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:58.848836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.65:42631 (local address 127.0.0.1:53684)
0929 03:19:58.849297 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0929 03:19:58.849337 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.178.65:42631: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":436}
W20260929 03:19:58.849529 1465 catalog_manager.cc:4584] TS 8e9d29d06b3b4e9cb85ad2c42b860961 (127.20.178.65:42631): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6287ccf63f744e43bb158f3ba1b68fd1: Network error: Client connection negotiation failed: client connection to 127.20.178.65:42631: connect: Connection refused (error 111)
I20260929 03:19:58.854694 21193 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuoaHbX/build/release/bin/kudu with pid 1813
I20260929 03:19:58.860380 21193 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuoaHbX/build/release/bin/kudu with pid 1445
I20260929 03:19:58.865978 21193 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260929 03:19:59.201212 21193 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260929 03:19:59.201297 21193 mini_ranger.cc:66] Stopping Ranger...
I20260929 03:19:59.545662 21193 mini_ranger.cc:68] Stopped Ranger
2026-09-29 03:19:59.545 UTC [512] LOG: received smart shutdown request
2026-09-29 03:19:59.549 UTC [512] LOG: background worker "logical replication launcher" (PID 521) exited with exit code 1
2026-09-29 03:19:59.549 UTC [516] LOG: shutting down
2026-09-29 03:19:59.714 UTC [512] LOG: database system is shut down
2026-09-29T03:19:59Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (45582 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651852476208-21193-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:19:59 dist-test-slave-c9qh krb5kdc[2013](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:19:59 dist-test-slave-c9qh krb5kdc[2013](info): set up 2 sockets
Sep 29 03:19:59 dist-test-slave-c9qh krb5kdc[2013](info): commencing operation
krb5kdc: starting...
W20260929 03:20:00.856527 21193 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651852476208-21193-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651852476208-21193-0/krb5kdc/test-user.keytab.
Sep 29 03:20:00 dist-test-slave-c9qh krb5kdc[2013](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652000, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:20:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:20:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.178.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.178.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.178.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.126.keytab.
Entry for principal kudu/127.20.178.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.126.keytab.
I20260929 03:20:00.976347 21193 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuoaHbX/build/release/bin/kudu
/tmp/dist-test-taskuoaHbX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651852476208-21193-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651852476208-21193-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651852476208-21193-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651852476208-21193-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.178.126:34487
--webserver_interface=127.20.178.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.126.keytab
--principal=kudu/127.20.178.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.178.84:44417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.178.126:34487,127.20.178.125:35687,127.20.178.124:44107
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651852476208-21193-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651852476208-21193-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651852476208-21193-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:20:01.056867 2026 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:20:01.057127 2026 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:20:01.057164 2026 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:20:01.058341 2026 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:20:01.058418 2026 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:20:01.058441 2026 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:20:01.058462 2026 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:20:01.058492 2026 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:20:01.060002 2026 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.178.84:44417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651852476208-21193-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651852476208-21193-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.178.126:34487,127.20.178.125:35687,127.20.178.124:44107
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.126.keytab
--principal=kudu/127.20.178.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.178.126:34487
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651852476208-21193-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.178.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2026
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651852476208-21193-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:20:01.060290 2026 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:20:01.060540 2026 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:20:01.065466 2034 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:20:01.065466 2031 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:20:01.065469 2032 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:20:01.065834 2026 server_base.cc:1034] running on GCE node
I20260929 03:20:01.066109 2026 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:20:01.066408 2026 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:20:01.067564 2026 hybrid_clock.cc:648] HybridClock initialized: now 1790652001067548 us; error 30 us; skew 500 ppm
Sep 29 03:20:01 dist-test-slave-c9qh krb5kdc[2013](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652001, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:20:01.074867 2026 init.cc:377] Logged in from keytab as kudu/127.20.178.126@KRBTEST.COM (short username kudu)
I20260929 03:20:01.075395 2026 webserver.cc:460] Webserver started at http://127.20.178.126:39437/ using document root <none> and password file <none>
I20260929 03:20:01.075592 2026 fs_manager.cc:359] Metadata directory not provided
I20260929 03:20:01.075635 2026 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:20:01.075728 2026 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:20:01.076840 2026 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651852476208-21193-0/minicluster-data/master-0/data/instance:
uuid: "417a2b170255454e947d7966ee192b21"
format_stamp: "Formatted at 2026-09-29 03:20:01 on dist-test-slave-c9qh"
server_key: "d52d64a941525f884a6568913873d279"
server_key_iv: "c84af9966d67e5f25a95a9dc89512ed4"
server_key_version: "encryptionkey@0"
I20260929 03:20:01.077167 2026 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651852476208-21193-0/minicluster-data/master-0/wal/instance:
uuid: "417a2b170255454e947d7966ee192b21"
format_stamp: "Formatted at 2026-09-29 03:20:01 on dist-test-slave-c9qh"
server_key: "d52d64a941525f884a6568913873d279"
server_key_iv: "c84af9966d67e5f25a95a9dc89512ed4"
server_key_version: "encryptionkey@0"
I20260929 03:20:01.078447 2026 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 03:20:01.079172 2041 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:20:01.079350 2026 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 03:20:01.079458 2026 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651852476208-21193-0/minicluster-data/master-0/data,/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651852476208-21193-0/minicluster-data/master-0/wal
uuid: "417a2b170255454e947d7966ee192b21"
format_stamp: "Formatted at 2026-09-29 03:20:01 on dist-test-slave-c9qh"
server_key: "d52d64a941525f884a6568913873d279"
server_key_iv: "c84af9966d67e5f25a95a9dc89512ed4"
server_key_version: "encryptionkey@0"
I20260929 03:20:01.079546 2026 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651852476208-21193-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651852476208-21193-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651852476208-21193-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:20:01.094686 2026 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:20:01.097136 2026 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:20:01.097308 2026 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:20:01.101465 2026 rpc_server.cc:307] RPC server started. Bound to: 127.20.178.126:34487
I20260929 03:20:01.101509 2093 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.178.126:34487 every 8 connection(s)
I20260929 03:20:01.101866 2026 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651852476208-21193-0/minicluster-data/master-0/data/info.pb
I20260929 03:20:01.102703 2094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:20:01.104180 2094 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.178.126" port: 34487 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 03:20:01 dist-test-slave-c9qh krb5kdc[2013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652001, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.126@KRBTEST.COM for kudu/127.20.178.126@KRBTEST.COM
I20260929 03:20:01.111085 21193 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuoaHbX/build/release/bin/kudu as pid 2026
I20260929 03:20:01.111191 21193 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651852476208-21193-0/minicluster-data/master-0/wal/instance
I20260929 03:20:01.111397 21193 external_mini_cluster.cc:1428] Setting key ff074e836b7875a2604f42bb1259f853
I20260929 03:20:01.113494 2097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:20:01.105011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47116 (local address 127.20.178.126:34487)
0929 03:20:01.105126 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0929 03:20:01.105131 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:20:01.105172 (+ 41us) server_negotiation.cc:407] Connection header received
0929 03:20:01.105607 (+ 435us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:20:01.105609 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:20:01.105625 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:20:01.105700 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:20:01.106212 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:20:01.106987 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:20:01.107825 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:20:01.108140 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:20:01.108215 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:20:01.110224 (+ 2009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:20:01.110230 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:20:01.110230 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:20:01.110236 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:20:01.111910 (+ 1674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:20:01.112268 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:20:01.112271 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:20:01.112271 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:20:01.112376 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:20:01.112583 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:20:01.112584 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:20:01.112584 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:20:01.112907 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:20:01.113015 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0929 03:20:01.113219 (+ 204us) server_negotiation.cc:299] Negotiation successful
0929 03:20:01.113317 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":85,"thread_start_us":60,"threads_started":1}
I20260929 03:20:01.113585 2096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:20:01.104861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:34487 (local address 127.0.0.1:47116)
0929 03:20:01.105004 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0929 03:20:01.105041 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 03:20:01.105481 (+ 440us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:20:01.105728 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:20:01.105730 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:20:01.105964 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:20:01.106133 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:20:01.106137 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:20:01.107091 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:20:01.107092 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:20:01.107755 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:20:01.107756 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:20:01.108212 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:20:01.108212 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:20:01.108267 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:20:01.108826 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:20:01.108834 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:20:01.110154 (+ 1320us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:20:01.111985 (+ 1831us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:20:01.111991 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:20:01.111992 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:20:01.112188 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:20:01.112441 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:20:01.112442 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:20:01.112442 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:20:01.112519 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:20:01.112991 (+ 472us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:20:01.112998 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:20:01.113075 (+ 77us) client_negotiation.cc:769] Sending connection context
0929 03:20:01.113186 (+ 111us) client_negotiation.cc:240] Negotiation successful
0929 03:20:01.113391 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"mutex_wait_us":45,"thread_start_us":63,"threads_started":1}
I20260929 03:20:01.114277 2094 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.178.125" port: 35687 } has no permanent_uuid. Determining permanent_uuid...
I20260929 03:20:01.114651 2096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:20:01.114521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.125:35687 (local address 127.0.0.1:35282)
0929 03:20:01.114571 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0929 03:20:01.114620 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.178.125:35687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260929 03:20:01.114805 2042 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.178.125:35687: connect: Connection refused (error 111)
I20260929 03:20:01.115275 2096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:20:01.115162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.125:35687 (local address 127.0.0.1:35286)
0929 03:20:01.115213 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0929 03:20:01.115245 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.178.125:35687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260929 03:20:01.115437 2094 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.178.125:35687: Network error: Client connection negotiation failed: client connection to 127.20.178.125:35687: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.20.178.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.178.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.178.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.125.keytab.
Entry for principal kudu/127.20.178.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.125.keytab.
I20260929 03:20:01.136003 21193 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuoaHbX/build/release/bin/kudu
/tmp/dist-test-taskuoaHbX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651852476208-21193-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651852476208-21193-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651852476208-21193-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651852476208-21193-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.178.125:35687
--webserver_interface=127.20.178.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.125.keytab
--principal=kudu/127.20.178.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.178.84:44417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.178.126:34487,127.20.178.125:35687,127.20.178.124:44107
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651852476208-21193-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651852476208-21193-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651852476208-21193-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 03:20:01.164713 2094 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.178.125" port: 35687 } attempt: 1
I20260929 03:20:01.165202 2096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:20:01.165069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.125:35687 (local address 127.0.0.1:35298)
0929 03:20:01.165132 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0929 03:20:01.165172 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.178.125:35687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260929 03:20:01.165557 2096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:20:01.165455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.125:35687 (local address 127.0.0.1:35310)
0929 03:20:01.165509 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0929 03:20:01.165539 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.178.125:35687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260929 03:20:01.165647 2094 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.178.125:35687: Network error: Client connection negotiation failed: client connection to 127.20.178.125:35687: connect: Connection refused (error 111)
W20260929 03:20:01.216940 2101 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:20:01.217202 2101 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:20:01.217242 2101 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:20:01.218408 2101 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:20:01.218485 2101 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:20:01.218513 2101 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:20:01.218545 2101 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:20:01.218569 2101 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:20:01.220047 2101 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.178.84:44417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651852476208-21193-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651852476208-21193-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.20.178.126:34487,127.20.178.125:35687,127.20.178.124:44107
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.125.keytab
--principal=kudu/127.20.178.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.178.125:35687
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651852476208-21193-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.20.178.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2101
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651852476208-21193-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:20:01.220299 2101 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:20:01.220479 2101 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:20:01.225296 2106 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:20:01.225339 2107 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:20:01.225312 2109 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:20:01.225608 2101 server_base.cc:1034] running on GCE node
I20260929 03:20:01.225862 2101 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:20:01.226162 2101 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:20:01.227331 2101 hybrid_clock.cc:648] HybridClock initialized: now 1790652001227328 us; error 43 us; skew 500 ppm
Sep 29 03:20:01 dist-test-slave-c9qh krb5kdc[2013](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652001, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:20:01.229812 2101 init.cc:377] Logged in from keytab as kudu/127.20.178.125@KRBTEST.COM (short username kudu)
I20260929 03:20:01.230319 2101 webserver.cc:460] Webserver started at http://127.20.178.125:43069/ using document root <none> and password file <none>
I20260929 03:20:01.230563 2101 fs_manager.cc:359] Metadata directory not provided
I20260929 03:20:01.230635 2101 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:20:01.230782 2101 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:20:01.231832 2101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651852476208-21193-0/minicluster-data/master-1/data/instance:
uuid: "33878b480d3344d4ad2b83be036bd4cd"
format_stamp: "Formatted at 2026-09-29 03:20:01 on dist-test-slave-c9qh"
server_key: "de19222df2fb451875639984d98576d9"
server_key_iv: "6dd5653563dc0092d88493076c67f44b"
server_key_version: "encryptionkey@0"
I20260929 03:20:01.232198 2101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651852476208-21193-0/minicluster-data/master-1/wal/instance:
uuid: "33878b480d3344d4ad2b83be036bd4cd"
format_stamp: "Formatted at 2026-09-29 03:20:01 on dist-test-slave-c9qh"
server_key: "de19222df2fb451875639984d98576d9"
server_key_iv: "6dd5653563dc0092d88493076c67f44b"
server_key_version: "encryptionkey@0"
I20260929 03:20:01.233610 2101 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 03:20:01.233796 2094 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.178.125" port: 35687 } attempt: 2
I20260929 03:20:01.234193 2096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:20:01.234079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.125:35687 (local address 127.0.0.1:35316)
0929 03:20:01.234135 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0929 03:20:01.234167 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.178.125:35687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260929 03:20:01.234560 2096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:20:01.234436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.125:35687 (local address 127.0.0.1:35330)
0929 03:20:01.234482 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0929 03:20:01.234528 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.178.125:35687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260929 03:20:01.234589 2116 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
W20260929 03:20:01.234683 2094 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.178.125:35687: Network error: Client connection negotiation failed: client connection to 127.20.178.125:35687: connect: Connection refused (error 111)
I20260929 03:20:01.234743 2101 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 03:20:01.234822 2101 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651852476208-21193-0/minicluster-data/master-1/data,/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651852476208-21193-0/minicluster-data/master-1/wal
uuid: "33878b480d3344d4ad2b83be036bd4cd"
format_stamp: "Formatted at 2026-09-29 03:20:01 on dist-test-slave-c9qh"
server_key: "de19222df2fb451875639984d98576d9"
server_key_iv: "6dd5653563dc0092d88493076c67f44b"
server_key_version: "encryptionkey@0"
I20260929 03:20:01.234913 2101 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651852476208-21193-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651852476208-21193-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651852476208-21193-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:20:01.255391 2101 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:20:01.257645 2101 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:20:01.257813 2101 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:20:01.261682 2101 rpc_server.cc:307] RPC server started. Bound to: 127.20.178.125:35687
I20260929 03:20:01.261718 2168 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.178.125:35687 every 8 connection(s)
I20260929 03:20:01.262133 2101 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651852476208-21193-0/minicluster-data/master-1/data/info.pb
I20260929 03:20:01.263345 2169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:20:01.265061 2169 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.178.126" port: 34487 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 03:20:01 dist-test-slave-c9qh krb5kdc[2013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652001, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.125@KRBTEST.COM for kudu/127.20.178.126@KRBTEST.COM
I20260929 03:20:01.271095 21193 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuoaHbX/build/release/bin/kudu as pid 2101
I20260929 03:20:01.271203 21193 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651852476208-21193-0/minicluster-data/master-1/wal/instance
I20260929 03:20:01.271394 21193 external_mini_cluster.cc:1428] Setting key f4330807d8d16f325f49b3aef3af5cf3
I20260929 03:20:01.274314 2097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:20:01.265811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47130 (local address 127.20.178.126:34487)
0929 03:20:01.265877 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0929 03:20:01.265879 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:20:01.266110 (+ 231us) server_negotiation.cc:407] Connection header received
0929 03:20:01.266536 (+ 426us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:20:01.266537 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:20:01.266547 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:20:01.266605 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:20:01.267176 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:20:01.267883 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:20:01.268911 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:20:01.269122 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:20:01.269183 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:20:01.271178 (+ 1995us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:20:01.271179 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:20:01.271180 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:20:01.271188 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:20:01.272918 (+ 1730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:20:01.273357 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:20:01.273359 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:20:01.273359 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:20:01.273418 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:20:01.273682 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:20:01.273683 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:20:01.273684 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:20:01.273803 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:20:01.273881 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0929 03:20:01.274180 (+ 299us) server_negotiation.cc:299] Negotiation successful
0929 03:20:01.274249 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260929 03:20:01.274605 2171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:20:01.265797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:34487 (local address 127.0.0.1:47130)
0929 03:20:01.265971 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0929 03:20:01.266014 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 03:20:01.266458 (+ 444us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:20:01.266655 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:20:01.266659 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:20:01.266904 (+ 245us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:20:01.267106 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:20:01.267110 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:20:01.267975 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:20:01.267976 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:20:01.268839 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:20:01.268842 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:20:01.269203 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:20:01.269203 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:20:01.269307 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:20:01.269752 (+ 445us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:20:01.269759 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:20:01.271087 (+ 1328us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:20:01.273018 (+ 1931us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:20:01.273026 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:20:01.273029 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:20:01.273274 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:20:01.273515 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:20:01.273516 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:20:01.273516 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:20:01.273612 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:20:01.273872 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:20:01.273878 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:20:01.274039 (+ 161us) client_negotiation.cc:769] Sending connection context
0929 03:20:01.274148 (+ 109us) client_negotiation.cc:240] Negotiation successful
0929 03:20:01.274319 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":100,"threads_started":1}
I20260929 03:20:01.275187 2169 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.178.125" port: 35687 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.20.178.124@KRBTEST.COM; defaulting to no policy
Sep 29 03:20:01 dist-test-slave-c9qh krb5kdc[2013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652001, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.125@KRBTEST.COM for kudu/127.20.178.125@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.178.124@KRBTEST.COM" created.
I20260929 03:20:01.283205 2171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:20:01.275468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.125:35687 (local address 127.0.0.1:35344)
0929 03:20:01.275523 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0929 03:20:01.275534 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 03:20:01.275570 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:20:01.275821 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:20:01.275822 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:20:01.275974 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:20:01.276055 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:20:01.276057 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:20:01.277445 (+ 1388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:20:01.277446 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:20:01.277925 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:20:01.277926 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:20:01.278586 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:20:01.278587 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:20:01.278637 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:20:01.279058 (+ 421us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:20:01.279062 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:20:01.280359 (+ 1297us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:20:01.282074 (+ 1715us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:20:01.282075 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:20:01.282075 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:20:01.282292 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:20:01.282501 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:20:01.282502 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:20:01.282503 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:20:01.282596 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:20:01.282938 (+ 342us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:20:01.282939 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:20:01.282988 (+ 49us) client_negotiation.cc:769] Sending connection context
0929 03:20:01.283064 (+ 76us) client_negotiation.cc:240] Negotiation successful
0929 03:20:01.283141 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260929 03:20:01.283397 2173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:20:01.275519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35344 (local address 127.20.178.125:35687)
0929 03:20:01.275675 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0929 03:20:01.275679 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:20:01.275697 (+ 18us) server_negotiation.cc:407] Connection header received
0929 03:20:01.275723 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:20:01.275725 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:20:01.275735 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:20:01.275796 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:20:01.276139 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:20:01.277346 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:20:01.278006 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:20:01.278497 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:20:01.278591 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:20:01.280431 (+ 1840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:20:01.280438 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:20:01.280438 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:20:01.280444 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:20:01.281990 (+ 1546us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:20:01.282362 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:20:01.282364 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:20:01.282365 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:20:01.282414 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:20:01.282669 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:20:01.282670 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:20:01.282670 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:20:01.282866 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:20:01.282949 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0929 03:20:01.283120 (+ 171us) server_negotiation.cc:299] Negotiation successful
0929 03:20:01.283204 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":102,"threads_started":1}
I20260929 03:20:01.283911 2169 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.178.124" port: 44107 } has no permanent_uuid. Determining permanent_uuid...
I20260929 03:20:01.284308 2171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:20:01.284174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.124:44107 (local address 127.0.0.1:41912)
0929 03:20:01.284230 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0929 03:20:01.284281 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.178.124:44107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260929 03:20:01.284438 2119 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.178.124:44107: connect: Connection refused (error 111)
I20260929 03:20:01.284871 2171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:20:01.284768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.124:44107 (local address 127.0.0.1:41920)
0929 03:20:01.284813 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0929 03:20:01.284843 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.178.124:44107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260929 03:20:01.284974 2169 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.178.124:44107: Network error: Client connection negotiation failed: client connection to 127.20.178.124:44107: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.178.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.124.keytab.
Entry for principal kudu/127.20.178.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.124.keytab.
I20260929 03:20:01.299294 21193 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuoaHbX/build/release/bin/kudu
/tmp/dist-test-taskuoaHbX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651852476208-21193-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651852476208-21193-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651852476208-21193-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651852476208-21193-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.178.124:44107
--webserver_interface=127.20.178.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.124.keytab
--principal=kudu/127.20.178.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.178.84:44417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.178.126:34487,127.20.178.125:35687,127.20.178.124:44107
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651852476208-21193-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651852476208-21193-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651852476208-21193-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 03:20:01.325855 2094 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.178.125" port: 35687 } attempt: 3
Sep 29 03:20:01 dist-test-slave-c9qh krb5kdc[2013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652001, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.126@KRBTEST.COM for kudu/127.20.178.125@KRBTEST.COM
I20260929 03:20:01.332928 2096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:20:01.326247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.125:35687 (local address 127.0.0.1:35358)
0929 03:20:01.326317 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0929 03:20:01.326343 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0929 03:20:01.326392 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:20:01.326542 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:20:01.326543 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:20:01.326715 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:20:01.326792 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:20:01.326793 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:20:01.327494 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:20:01.327495 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:20:01.328022 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:20:01.328024 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:20:01.328432 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:20:01.328433 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:20:01.328517 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:20:01.329008 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:20:01.329021 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:20:01.330206 (+ 1185us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:20:01.331860 (+ 1654us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:20:01.331860 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:20:01.331861 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:20:01.332041 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:20:01.332247 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:20:01.332248 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:20:01.332248 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:20:01.332314 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:20:01.332571 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:20:01.332572 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:20:01.332622 (+ 50us) client_negotiation.cc:769] Sending connection context
0929 03:20:01.332709 (+ 87us) client_negotiation.cc:240] Negotiation successful
0929 03:20:01.332810 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260929 03:20:01.332921 2173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:20:01.326302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35358 (local address 127.20.178.125:35687)
0929 03:20:01.326364 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0929 03:20:01.326366 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:20:01.326384 (+ 18us) server_negotiation.cc:407] Connection header received
0929 03:20:01.326450 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:20:01.326451 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:20:01.326470 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:20:01.326536 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:20:01.326877 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:20:01.327405 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:20:01.328109 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:20:01.328321 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:20:01.328432 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:20:01.330293 (+ 1861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:20:01.330295 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:20:01.330296 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:20:01.330303 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:20:01.331757 (+ 1454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:20:01.332115 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:20:01.332116 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:20:01.332117 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:20:01.332179 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:20:01.332385 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:20:01.332385 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:20:01.332386 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:20:01.332498 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:20:01.332580 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 03:20:01.332731 (+ 151us) server_negotiation.cc:299] Negotiation successful
0929 03:20:01.332810 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260929 03:20:01.333411 2094 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.178.124" port: 44107 } has no permanent_uuid. Determining permanent_uuid...
I20260929 03:20:01.333779 2096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:20:01.333676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.124:44107 (local address 127.0.0.1:41926)
0929 03:20:01.333724 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0929 03:20:01.333755 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.178.124:44107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260929 03:20:01.334081 2096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:20:01.334024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.124:44107 (local address 127.0.0.1:41930)
0929 03:20:01.334048 (+ 24us) negotiation.cc:107] Waiting for socket to connect
0929 03:20:01.334068 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.178.124:44107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
W20260929 03:20:01.334198 2094 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.178.124:44107: Network error: Client connection negotiation failed: client connection to 127.20.178.124:44107: connect: Connection refused (error 111)
I20260929 03:20:01.334194 2169 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.178.124" port: 44107 } attempt: 1
I20260929 03:20:01.334554 2171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:20:01.334442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.124:44107 (local address 127.0.0.1:41938)
0929 03:20:01.334493 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0929 03:20:01.334526 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.178.124:44107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260929 03:20:01.334848 2171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:20:01.334788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.124:44107 (local address 127.0.0.1:41946)
0929 03:20:01.334810 (+ 22us) negotiation.cc:107] Waiting for socket to connect
0929 03:20:01.334832 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.178.124:44107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":12}
W20260929 03:20:01.334939 2169 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.178.124:44107: Network error: Client connection negotiation failed: client connection to 127.20.178.124:44107: connect: Connection refused (error 111)
I20260929 03:20:01.365373 2094 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.178.124" port: 44107 } attempt: 1
I20260929 03:20:01.365896 2096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:20:01.365749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.124:44107 (local address 127.0.0.1:41954)
0929 03:20:01.365823 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0929 03:20:01.365862 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.178.124:44107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260929 03:20:01.366325 2096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:20:01.366192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.124:44107 (local address 127.0.0.1:41966)
0929 03:20:01.366253 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0929 03:20:01.366296 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.178.124:44107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260929 03:20:01.366449 2094 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.178.124:44107: Network error: Client connection negotiation failed: client connection to 127.20.178.124:44107: connect: Connection refused (error 111)
W20260929 03:20:01.381932 2176 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:20:01.382156 2176 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:20:01.382180 2176 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:20:01.383352 2176 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:20:01.383395 2176 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:20:01.383410 2176 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:20:01.383423 2176 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:20:01.383435 2176 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:20:01.384953 2176 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.178.84:44417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651852476208-21193-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651852476208-21193-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.20.178.126:34487,127.20.178.125:35687,127.20.178.124:44107
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.124.keytab
--principal=kudu/127.20.178.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.178.124:44107
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651852476208-21193-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.20.178.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2176
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651852476208-21193-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:20:01.385190 2176 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:20:01.385367 2176 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:20:01.390110 2181 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:20:01.390115 2182 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:20:01.390115 2184 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:20:01.390368 2176 server_base.cc:1034] running on GCE node
I20260929 03:20:01.390645 2176 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:20:01.390942 2176 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:20:01.392115 2176 hybrid_clock.cc:648] HybridClock initialized: now 1790652001392101 us; error 42 us; skew 500 ppm
Sep 29 03:20:01 dist-test-slave-c9qh krb5kdc[2013](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652001, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:20:01.394718 2176 init.cc:377] Logged in from keytab as kudu/127.20.178.124@KRBTEST.COM (short username kudu)
I20260929 03:20:01.395162 2176 webserver.cc:460] Webserver started at http://127.20.178.124:42563/ using document root <none> and password file <none>
I20260929 03:20:01.395380 2176 fs_manager.cc:359] Metadata directory not provided
I20260929 03:20:01.395443 2176 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:20:01.395589 2176 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:20:01.396710 2176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651852476208-21193-0/minicluster-data/master-2/data/instance:
uuid: "9628e08b2c2c484bba6eecf4387c7ce6"
format_stamp: "Formatted at 2026-09-29 03:20:01 on dist-test-slave-c9qh"
server_key: "34bf125badd9cd4a941bddb553086799"
server_key_iv: "40eb43b3e13f9cb32c1f5264593fa053"
server_key_version: "encryptionkey@0"
I20260929 03:20:01.397059 2176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651852476208-21193-0/minicluster-data/master-2/wal/instance:
uuid: "9628e08b2c2c484bba6eecf4387c7ce6"
format_stamp: "Formatted at 2026-09-29 03:20:01 on dist-test-slave-c9qh"
server_key: "34bf125badd9cd4a941bddb553086799"
server_key_iv: "40eb43b3e13f9cb32c1f5264593fa053"
server_key_version: "encryptionkey@0"
I20260929 03:20:01.398348 2176 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260929 03:20:01.399025 2191 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:20:01.399189 2176 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 03:20:01.399294 2176 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651852476208-21193-0/minicluster-data/master-2/data,/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651852476208-21193-0/minicluster-data/master-2/wal
uuid: "9628e08b2c2c484bba6eecf4387c7ce6"
format_stamp: "Formatted at 2026-09-29 03:20:01 on dist-test-slave-c9qh"
server_key: "34bf125badd9cd4a941bddb553086799"
server_key_iv: "40eb43b3e13f9cb32c1f5264593fa053"
server_key_version: "encryptionkey@0"
I20260929 03:20:01.399364 2176 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651852476208-21193-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651852476208-21193-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651852476208-21193-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:20:01.403105 2169 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.178.124" port: 44107 } attempt: 2
I20260929 03:20:01.403534 2171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:20:01.403407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.124:44107 (local address 127.0.0.1:41972)
0929 03:20:01.403465 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0929 03:20:01.403505 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.178.124:44107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260929 03:20:01.403877 2171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:20:01.403779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.124:44107 (local address 127.0.0.1:41976)
0929 03:20:01.403826 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0929 03:20:01.403852 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.178.124:44107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260929 03:20:01.404006 2169 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.178.124:44107: Network error: Client connection negotiation failed: client connection to 127.20.178.124:44107: connect: Connection refused (error 111)
I20260929 03:20:01.423573 2176 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:20:01.425976 2176 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:20:01.426090 2176 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:20:01.429929 2176 rpc_server.cc:307] RPC server started. Bound to: 127.20.178.124:44107
I20260929 03:20:01.430006 2243 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.178.124:44107 every 8 connection(s)
I20260929 03:20:01.430332 2176 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651852476208-21193-0/minicluster-data/master-2/data/info.pb
I20260929 03:20:01.431195 2244 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:20:01.432772 2244 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.178.126" port: 34487 } has no permanent_uuid. Determining permanent_uuid...
I20260929 03:20:01.434487 21193 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuoaHbX/build/release/bin/kudu as pid 2176
I20260929 03:20:01.434581 21193 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651852476208-21193-0/minicluster-data/master-2/wal/instance
I20260929 03:20:01.434751 21193 external_mini_cluster.cc:1428] Setting key 1e95387187f3e760be31f79f79224db3
Sep 29 03:20:01 dist-test-slave-c9qh krb5kdc[2013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652001, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.124@KRBTEST.COM for kudu/127.20.178.126@KRBTEST.COM
Sep 29 03:20:01 dist-test-slave-c9qh krb5kdc[2013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652000, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.178.124@KRBTEST.COM
I20260929 03:20:01.441605 2094 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.178.124" port: 44107 } attempt: 2
I20260929 03:20:01.443081 2097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:20:01.433508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47134 (local address 127.20.178.126:34487)
0929 03:20:01.433580 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0929 03:20:01.433582 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:20:01.433750 (+ 168us) server_negotiation.cc:407] Connection header received
0929 03:20:01.434189 (+ 439us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:20:01.434189 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:20:01.434198 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:20:01.434259 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:20:01.434937 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:20:01.435698 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:20:01.436954 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:20:01.437138 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:20:01.437195 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:20:01.439519 (+ 2324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:20:01.439521 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:20:01.439521 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:20:01.439529 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:20:01.441197 (+ 1668us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:20:01.442011 (+ 814us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:20:01.442011 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:20:01.442012 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:20:01.442094 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:20:01.442385 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:20:01.442386 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:20:01.442387 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:20:01.442519 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:20:01.442595 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0929 03:20:01.442886 (+ 291us) server_negotiation.cc:299] Negotiation successful
0929 03:20:01.442974 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260929 03:20:01.443332 2246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:20:01.433448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:34487 (local address 127.0.0.1:47134)
0929 03:20:01.433620 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0929 03:20:01.433661 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 03:20:01.434106 (+ 445us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:20:01.434334 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:20:01.434337 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:20:01.434571 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:20:01.434830 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:20:01.434836 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:20:01.435777 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:20:01.435778 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:20:01.436873 (+ 1095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:20:01.436876 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:20:01.437210 (+ 334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:20:01.437211 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:20:01.437358 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:20:01.437925 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:20:01.437934 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:20:01.439431 (+ 1497us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:20:01.441293 (+ 1862us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:20:01.441300 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:20:01.441302 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:20:01.441530 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:20:01.442166 (+ 636us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:20:01.442167 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:20:01.442168 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:20:01.442319 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:20:01.442586 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:20:01.442594 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:20:01.442743 (+ 149us) client_negotiation.cc:769] Sending connection context
0929 03:20:01.442851 (+ 108us) client_negotiation.cc:240] Negotiation successful
0929 03:20:01.443070 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":94,"threads_started":1}
I20260929 03:20:01.443879 2244 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.178.125" port: 35687 } has no permanent_uuid. Determining permanent_uuid...
I20260929 03:20:01.444950 2248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:20:01.435146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41978 (local address 127.20.178.124:44107)
0929 03:20:01.435285 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0929 03:20:01.435289 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:20:01.435328 (+ 39us) server_negotiation.cc:407] Connection header received
0929 03:20:01.435396 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:20:01.435397 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:20:01.435412 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:20:01.435471 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:20:01.436157 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:20:01.437195 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:20:01.438053 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:20:01.438470 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:20:01.438588 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:20:01.441212 (+ 2624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:20:01.441221 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:20:01.441221 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:20:01.441227 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:20:01.443181 (+ 1954us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:20:01.443637 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:20:01.443640 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:20:01.443641 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:20:01.443773 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:20:01.444017 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:20:01.444019 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:20:01.444019 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:20:01.444189 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:20:01.444351 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0929 03:20:01.444776 (+ 425us) server_negotiation.cc:299] Negotiation successful
0929 03:20:01.444849 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":87,"threads_started":1}
Sep 29 03:20:01 dist-test-slave-c9qh krb5kdc[2013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652001, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.126@KRBTEST.COM for kudu/127.20.178.124@KRBTEST.COM
Sep 29 03:20:01 dist-test-slave-c9qh krb5kdc[2013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652001, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.124@KRBTEST.COM for kudu/127.20.178.125@KRBTEST.COM
I20260929 03:20:01.450752 2096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:20:01.441871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.124:44107 (local address 127.0.0.1:41984)
0929 03:20:01.441919 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0929 03:20:01.441928 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 03:20:01.441966 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:20:01.442229 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:20:01.442230 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:20:01.442385 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:20:01.442433 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:20:01.442435 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:20:01.443802 (+ 1367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:20:01.443803 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:20:01.444559 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:20:01.444561 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:20:01.445079 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:20:01.445080 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:20:01.445134 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:20:01.445559 (+ 425us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:20:01.445564 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:20:01.447055 (+ 1491us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:20:01.449697 (+ 2642us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:20:01.449699 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:20:01.449699 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:20:01.449859 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:20:01.450068 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:20:01.450068 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:20:01.450069 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:20:01.450119 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:20:01.450393 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:20:01.450394 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:20:01.450442 (+ 48us) client_negotiation.cc:769] Sending connection context
0929 03:20:01.450525 (+ 83us) client_negotiation.cc:240] Negotiation successful
0929 03:20:01.450625 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260929 03:20:01.451133 2249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:20:01.441938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41984 (local address 127.20.178.124:44107)
0929 03:20:01.442104 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0929 03:20:01.442106 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:20:01.442113 (+ 7us) server_negotiation.cc:407] Connection header received
0929 03:20:01.442133 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:20:01.442133 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:20:01.442144 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:20:01.442506 (+ 362us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:20:01.442535 (+ 29us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:20:01.443725 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:20:01.444706 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:20:01.444983 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:20:01.445284 (+ 301us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:20:01.447145 (+ 1861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:20:01.447146 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:20:01.447146 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:20:01.447152 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:20:01.449614 (+ 2462us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:20:01.449934 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:20:01.449936 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:20:01.449936 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:20:01.449998 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:20:01.450182 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:20:01.450183 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:20:01.450183 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:20:01.450319 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:20:01.450387 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0929 03:20:01.450945 (+ 558us) server_negotiation.cc:299] Negotiation successful
0929 03:20:01.451020 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":350,"threads_started":1}
I20260929 03:20:01.451508 2173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:20:01.444195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35372 (local address 127.20.178.125:35687)
0929 03:20:01.444237 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0929 03:20:01.444239 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:20:01.444247 (+ 8us) server_negotiation.cc:407] Connection header received
0929 03:20:01.444314 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:20:01.444315 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:20:01.444324 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:20:01.444377 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:20:01.444653 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:20:01.445325 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:20:01.446098 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:20:01.446402 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:20:01.446466 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:20:01.448544 (+ 2078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:20:01.448545 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:20:01.448546 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:20:01.448553 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:20:01.450141 (+ 1588us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:20:01.450489 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:20:01.450491 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:20:01.450491 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:20:01.450548 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:20:01.450760 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:20:01.450761 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:20:01.450761 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:20:01.450908 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:20:01.450977 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0929 03:20:01.451334 (+ 357us) server_negotiation.cc:299] Negotiation successful
0929 03:20:01.451418 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260929 03:20:01.451762 2246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:20:01.444141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.125:35687 (local address 127.0.0.1:35372)
0929 03:20:01.444205 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0929 03:20:01.444215 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 03:20:01.444251 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:20:01.444378 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:20:01.444379 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:20:01.444508 (+ 129us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:20:01.444550 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:20:01.444551 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:20:01.445416 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:20:01.445417 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:20:01.446023 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:20:01.446027 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:20:01.446463 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:20:01.446463 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:20:01.446539 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:20:01.446945 (+ 406us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:20:01.446949 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:20:01.448444 (+ 1495us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:20:01.450238 (+ 1794us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:20:01.450238 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:20:01.450238 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:20:01.450423 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:20:01.450629 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:20:01.450630 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:20:01.450630 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:20:01.450689 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:20:01.451010 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:20:01.451011 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:20:01.451058 (+ 47us) client_negotiation.cc:769] Sending connection context
0929 03:20:01.451128 (+ 70us) client_negotiation.cc:240] Negotiation successful
0929 03:20:01.451671 (+ 543us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
Sep 29 03:20:01 dist-test-slave-c9qh krb5kdc[2013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652000, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.178.126@KRBTEST.COM
I20260929 03:20:01.453392 2094 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 417a2b170255454e947d7966ee192b21: Bootstrap starting.
I20260929 03:20:01.454211 2094 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 417a2b170255454e947d7966ee192b21: Neither blocks nor log segments found. Creating new log.
I20260929 03:20:01.454551 2094 log.cc:824] T 00000000000000000000000000000000 P 417a2b170255454e947d7966ee192b21: Log is configured to *not* fsync() on all Append() calls
I20260929 03:20:01.454967 2097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:20:01.445992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47140 (local address 127.20.178.126:34487)
0929 03:20:01.446027 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0929 03:20:01.446029 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:20:01.446038 (+ 9us) server_negotiation.cc:407] Connection header received
0929 03:20:01.446515 (+ 477us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:20:01.446516 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:20:01.446528 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:20:01.446585 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:20:01.447190 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:20:01.447793 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:20:01.448532 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:20:01.448832 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:20:01.448891 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:20:01.452243 (+ 3352us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:20:01.452245 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:20:01.452245 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:20:01.452251 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:20:01.453811 (+ 1560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:20:01.454146 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:20:01.454147 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:20:01.454147 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:20:01.454205 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:20:01.454417 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:20:01.454417 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:20:01.454418 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:20:01.454558 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:20:01.454687 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0929 03:20:01.454796 (+ 109us) server_negotiation.cc:299] Negotiation successful
0929 03:20:01.454873 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260929 03:20:01.455117 2244 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.178.124" port: 44107 } has no permanent_uuid. Determining permanent_uuid...
I20260929 03:20:01.455739 2094 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 417a2b170255454e947d7966ee192b21: No bootstrap required, opened a new log
I20260929 03:20:01.457734 2094 raft_consensus.cc:348] T 00000000000000000000000000000000 P 417a2b170255454e947d7966ee192b21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "417a2b170255454e947d7966ee192b21" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 34487 } } peers { permanent_uuid: "33878b480d3344d4ad2b83be036bd4cd" member_type: VOTER last_known_addr { host: "127.20.178.125" port: 35687 } } peers { permanent_uuid: "9628e08b2c2c484bba6eecf4387c7ce6" member_type: VOTER last_known_addr { host: "127.20.178.124" port: 44107 } }
I20260929 03:20:01.457894 2094 raft_consensus.cc:374] T 00000000000000000000000000000000 P 417a2b170255454e947d7966ee192b21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:20:01.457944 2094 raft_consensus.cc:729] T 00000000000000000000000000000000 P 417a2b170255454e947d7966ee192b21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 417a2b170255454e947d7966ee192b21, State: Initialized, Role: FOLLOWER
I20260929 03:20:01.458057 2094 consensus_queue.cc:260] T 00000000000000000000000000000000 P 417a2b170255454e947d7966ee192b21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "417a2b170255454e947d7966ee192b21" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 34487 } } peers { permanent_uuid: "33878b480d3344d4ad2b83be036bd4cd" member_type: VOTER last_known_addr { host: "127.20.178.125" port: 35687 } } peers { permanent_uuid: "9628e08b2c2c484bba6eecf4387c7ce6" member_type: VOTER last_known_addr { host: "127.20.178.124" port: 44107 } }
I20260929 03:20:01.458365 2251 sys_catalog.cc:455] T 00000000000000000000000000000000 P 417a2b170255454e947d7966ee192b21 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "417a2b170255454e947d7966ee192b21" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 34487 } } peers { permanent_uuid: "33878b480d3344d4ad2b83be036bd4cd" member_type: VOTER last_known_addr { host: "127.20.178.125" port: 35687 } } peers { permanent_uuid: "9628e08b2c2c484bba6eecf4387c7ce6" member_type: VOTER last_known_addr { host: "127.20.178.124" port: 44107 } } }
I20260929 03:20:01.458413 2094 sys_catalog.cc:564] T 00000000000000000000000000000000 P 417a2b170255454e947d7966ee192b21 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:20:01.458877 2251 sys_catalog.cc:458] T 00000000000000000000000000000000 P 417a2b170255454e947d7966ee192b21 [sys.catalog]: This master's current role is: FOLLOWER
Sep 29 03:20:01 dist-test-slave-c9qh krb5kdc[2013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652001, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.124@KRBTEST.COM for kudu/127.20.178.124@KRBTEST.COM
W20260929 03:20:01.460788 2262 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 417a2b170255454e947d7966ee192b21: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 03:20:01.460844 2262 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
Sep 29 03:20:01 dist-test-slave-c9qh krb5kdc[2013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652000, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.178.125@KRBTEST.COM
I20260929 03:20:01.464257 2246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:20:01.455392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.124:44107 (local address 127.0.0.1:41992)
0929 03:20:01.455435 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0929 03:20:01.455446 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 03:20:01.455480 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:20:01.456835 (+ 1355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:20:01.456835 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:20:01.456963 (+ 128us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:20:01.456990 (+ 27us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:20:01.456992 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:20:01.457643 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:20:01.457644 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:20:01.458250 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:20:01.458264 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:20:01.458734 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:20:01.458735 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:20:01.458796 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:20:01.459193 (+ 397us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:20:01.459199 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:20:01.461347 (+ 2148us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:20:01.463221 (+ 1874us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:20:01.463222 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:20:01.463222 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:20:01.463409 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:20:01.463581 (+ 172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:20:01.463582 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:20:01.463582 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:20:01.463630 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:20:01.463961 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:20:01.463962 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:20:01.464007 (+ 45us) client_negotiation.cc:769] Sending connection context
0929 03:20:01.464066 (+ 59us) client_negotiation.cc:240] Negotiation successful
0929 03:20:01.464157 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260929 03:20:01.461615 2026 master_runner.cc:186] Error getting master registration for 127.20.178.125:35687: OK, instance_id { permanent_uuid: "33878b480d3344d4ad2b83be036bd4cd" instance_seqno: 1790652001260496 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 03:20:01.464754 2026 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 03:20:01.464838 2249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:20:01.455472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41992 (local address 127.20.178.124:44107)
0929 03:20:01.456719 (+ 1247us) server_negotiation.cc:206] Beginning negotiation
0929 03:20:01.456721 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:20:01.456729 (+ 8us) server_negotiation.cc:407] Connection header received
0929 03:20:01.456748 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:20:01.456749 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:20:01.456765 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:20:01.456832 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:20:01.457063 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:20:01.457550 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:20:01.458352 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:20:01.458652 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:20:01.458720 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:20:01.461432 (+ 2712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:20:01.461433 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:20:01.461433 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:20:01.461440 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:20:01.463132 (+ 1692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:20:01.463478 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:20:01.463479 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:20:01.463479 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:20:01.463525 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:20:01.463688 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:20:01.463689 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:20:01.463690 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:20:01.463864 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:20:01.463940 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0929 03:20:01.464657 (+ 717us) server_negotiation.cc:299] Negotiation successful
0929 03:20:01.464741 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1226}
I20260929 03:20:01.465723 2173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:20:01.455605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35388 (local address 127.20.178.125:35687)
0929 03:20:01.455643 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0929 03:20:01.455645 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:20:01.455653 (+ 8us) server_negotiation.cc:407] Connection header received
0929 03:20:01.455673 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:20:01.455673 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:20:01.455683 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:20:01.455737 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:20:01.456208 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:20:01.456914 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:20:01.457868 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:20:01.458156 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:20:01.458206 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:20:01.462173 (+ 3967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:20:01.462175 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:20:01.462175 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:20:01.462182 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:20:01.464140 (+ 1958us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:20:01.464521 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:20:01.464521 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:20:01.464522 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:20:01.464578 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:20:01.464825 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:20:01.464826 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:20:01.464826 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:20:01.464955 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:20:01.465133 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0929 03:20:01.465552 (+ 419us) server_negotiation.cc:299] Negotiation successful
0929 03:20:01.465629 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260929 03:20:01.466094 2244 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9628e08b2c2c484bba6eecf4387c7ce6: Bootstrap starting.
I20260929 03:20:01.466706 2244 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9628e08b2c2c484bba6eecf4387c7ce6: Neither blocks nor log segments found. Creating new log.
I20260929 03:20:01.466985 2244 log.cc:824] T 00000000000000000000000000000000 P 9628e08b2c2c484bba6eecf4387c7ce6: Log is configured to *not* fsync() on all Append() calls
I20260929 03:20:01.467741 2244 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9628e08b2c2c484bba6eecf4387c7ce6: No bootstrap required, opened a new log
I20260929 03:20:01.468984 21193 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 03:20:01.469646 2244 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9628e08b2c2c484bba6eecf4387c7ce6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "417a2b170255454e947d7966ee192b21" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 34487 } } peers { permanent_uuid: "33878b480d3344d4ad2b83be036bd4cd" member_type: VOTER last_known_addr { host: "127.20.178.125" port: 35687 } } peers { permanent_uuid: "9628e08b2c2c484bba6eecf4387c7ce6" member_type: VOTER last_known_addr { host: "127.20.178.124" port: 44107 } }
I20260929 03:20:01.469791 2244 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9628e08b2c2c484bba6eecf4387c7ce6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:20:01.469834 2244 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9628e08b2c2c484bba6eecf4387c7ce6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9628e08b2c2c484bba6eecf4387c7ce6, State: Initialized, Role: FOLLOWER
I20260929 03:20:01.469959 2244 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9628e08b2c2c484bba6eecf4387c7ce6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "417a2b170255454e947d7966ee192b21" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 34487 } } peers { permanent_uuid: "33878b480d3344d4ad2b83be036bd4cd" member_type: VOTER last_known_addr { host: "127.20.178.125" port: 35687 } } peers { permanent_uuid: "9628e08b2c2c484bba6eecf4387c7ce6" member_type: VOTER last_known_addr { host: "127.20.178.124" port: 44107 } }
I20260929 03:20:01.470306 2244 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9628e08b2c2c484bba6eecf4387c7ce6 [sys.catalog]: configured and running, proceeding with master startup.
W20260929 03:20:01.475847 2283 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9628e08b2c2c484bba6eecf4387c7ce6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 03:20:01.477115 2283 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 03:20:01.473632 2176 master_runner.cc:186] Error getting master registration for 127.20.178.125:35687: OK, instance_id { permanent_uuid: "33878b480d3344d4ad2b83be036bd4cd" instance_seqno: 1790652001260496 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 03:20:01.477320 2176 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 03:20:01.476814 2268 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9628e08b2c2c484bba6eecf4387c7ce6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "417a2b170255454e947d7966ee192b21" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 34487 } } peers { permanent_uuid: "33878b480d3344d4ad2b83be036bd4cd" member_type: VOTER last_known_addr { host: "127.20.178.125" port: 35687 } } peers { permanent_uuid: "9628e08b2c2c484bba6eecf4387c7ce6" member_type: VOTER last_known_addr { host: "127.20.178.124" port: 44107 } } }
I20260929 03:20:01.477384 2268 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9628e08b2c2c484bba6eecf4387c7ce6 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 03:20:01.479074 2173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:20:01.470599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35394 (local address 127.20.178.125:35687)
0929 03:20:01.470662 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0929 03:20:01.470664 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:20:01.470814 (+ 150us) server_negotiation.cc:407] Connection header received
0929 03:20:01.470908 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:20:01.470909 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:20:01.470919 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:20:01.470950 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:20:01.471797 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:20:01.472530 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:20:01.473781 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:20:01.474083 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:20:01.474125 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:20:01.475988 (+ 1863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:20:01.475990 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:20:01.475990 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:20:01.475997 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:20:01.477753 (+ 1756us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:20:01.478158 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:20:01.478159 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:20:01.478159 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:20:01.478223 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:20:01.478436 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:20:01.478437 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:20:01.478438 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:20:01.478580 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:20:01.478674 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0929 03:20:01.478874 (+ 200us) server_negotiation.cc:299] Negotiation successful
0929 03:20:01.478957 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260929 03:20:01.479984 2097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:20:01.470800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47152 (local address 127.20.178.126:34487)
0929 03:20:01.470853 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0929 03:20:01.470855 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:20:01.471903 (+ 1048us) server_negotiation.cc:407] Connection header received
0929 03:20:01.471983 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:20:01.471984 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:20:01.471995 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:20:01.472039 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:20:01.472915 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:20:01.473561 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:20:01.474605 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:20:01.474892 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:20:01.474933 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:20:01.476850 (+ 1917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:20:01.476851 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:20:01.476851 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:20:01.476858 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:20:01.478421 (+ 1563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:20:01.478777 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:20:01.478778 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:20:01.478778 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:20:01.478833 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:20:01.479035 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:20:01.479035 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:20:01.479036 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:20:01.479156 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:20:01.479249 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0929 03:20:01.479840 (+ 591us) server_negotiation.cc:299] Negotiation successful
0929 03:20:01.479926 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260929 03:20:01.484998 2249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:20:01.470711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42008 (local address 127.20.178.124:44107)
0929 03:20:01.470748 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0929 03:20:01.470750 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:20:01.477715 (+ 6965us) server_negotiation.cc:407] Connection header received
0929 03:20:01.477819 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:20:01.477820 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:20:01.477830 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:20:01.477860 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:20:01.478459 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:20:01.479170 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:20:01.480252 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:20:01.480542 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:20:01.480580 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:20:01.482031 (+ 1451us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:20:01.482033 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:20:01.482033 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:20:01.482041 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:20:01.483728 (+ 1687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:20:01.484126 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:20:01.484127 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:20:01.484128 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:20:01.484188 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:20:01.484400 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:20:01.484401 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:20:01.484401 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:20:01.484533 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:20:01.484678 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0929 03:20:01.484827 (+ 149us) server_negotiation.cc:299] Negotiation successful
0929 03:20:01.484905 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260929 03:20:01.495184 2169 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.178.124" port: 44107 } attempt: 3
Sep 29 03:20:01 dist-test-slave-c9qh krb5kdc[2013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652001, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.125@KRBTEST.COM for kudu/127.20.178.124@KRBTEST.COM
I20260929 03:20:01.503851 2171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:20:01.495458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.124:44107 (local address 127.0.0.1:42012)
0929 03:20:01.495505 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0929 03:20:01.495517 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 03:20:01.495552 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:20:01.495824 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:20:01.495825 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:20:01.496018 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:20:01.496070 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:20:01.496072 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:20:01.497010 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:20:01.497011 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:20:01.497832 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:20:01.497835 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:20:01.498307 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:20:01.498308 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:20:01.498379 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:20:01.498889 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:20:01.498894 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:20:01.500425 (+ 1531us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:20:01.502748 (+ 2323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:20:01.502749 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:20:01.502750 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:20:01.502969 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:20:01.503188 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:20:01.503190 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:20:01.503190 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:20:01.503253 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:20:01.503540 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:20:01.503541 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:20:01.503586 (+ 45us) client_negotiation.cc:769] Sending connection context
0929 03:20:01.503647 (+ 61us) client_negotiation.cc:240] Negotiation successful
0929 03:20:01.503745 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260929 03:20:01.504395 2249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:20:01.495658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42012 (local address 127.20.178.124:44107)
0929 03:20:01.495705 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0929 03:20:01.495707 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:20:01.495713 (+ 6us) server_negotiation.cc:407] Connection header received
0929 03:20:01.495732 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:20:01.495733 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:20:01.495750 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:20:01.495799 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:20:01.496154 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:20:01.496924 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:20:01.497917 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:20:01.498228 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:20:01.498299 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:20:01.500514 (+ 2215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:20:01.500516 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:20:01.500516 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:20:01.500523 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:20:01.502658 (+ 2135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:20:01.503046 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:20:01.503047 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:20:01.503047 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:20:01.503107 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:20:01.503318 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:20:01.503319 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:20:01.503319 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:20:01.503456 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:20:01.503515 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0929 03:20:01.504220 (+ 705us) server_negotiation.cc:299] Negotiation successful
0929 03:20:01.504295 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260929 03:20:01.506186 2169 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33878b480d3344d4ad2b83be036bd4cd: Bootstrap starting.
I20260929 03:20:01.506950 2169 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 33878b480d3344d4ad2b83be036bd4cd: Neither blocks nor log segments found. Creating new log.
I20260929 03:20:01.507280 2169 log.cc:824] T 00000000000000000000000000000000 P 33878b480d3344d4ad2b83be036bd4cd: Log is configured to *not* fsync() on all Append() calls
I20260929 03:20:01.508687 2169 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33878b480d3344d4ad2b83be036bd4cd: No bootstrap required, opened a new log
I20260929 03:20:01.510619 2169 raft_consensus.cc:348] T 00000000000000000000000000000000 P 33878b480d3344d4ad2b83be036bd4cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "417a2b170255454e947d7966ee192b21" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 34487 } } peers { permanent_uuid: "33878b480d3344d4ad2b83be036bd4cd" member_type: VOTER last_known_addr { host: "127.20.178.125" port: 35687 } } peers { permanent_uuid: "9628e08b2c2c484bba6eecf4387c7ce6" member_type: VOTER last_known_addr { host: "127.20.178.124" port: 44107 } }
I20260929 03:20:01.510759 2169 raft_consensus.cc:374] T 00000000000000000000000000000000 P 33878b480d3344d4ad2b83be036bd4cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:20:01.510792 2169 raft_consensus.cc:729] T 00000000000000000000000000000000 P 33878b480d3344d4ad2b83be036bd4cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33878b480d3344d4ad2b83be036bd4cd, State: Initialized, Role: FOLLOWER
I20260929 03:20:01.510912 2169 consensus_queue.cc:260] T 00000000000000000000000000000000 P 33878b480d3344d4ad2b83be036bd4cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "417a2b170255454e947d7966ee192b21" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 34487 } } peers { permanent_uuid: "33878b480d3344d4ad2b83be036bd4cd" member_type: VOTER last_known_addr { host: "127.20.178.125" port: 35687 } } peers { permanent_uuid: "9628e08b2c2c484bba6eecf4387c7ce6" member_type: VOTER last_known_addr { host: "127.20.178.124" port: 44107 } }
I20260929 03:20:01.511229 2169 sys_catalog.cc:564] T 00000000000000000000000000000000 P 33878b480d3344d4ad2b83be036bd4cd [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:20:01.511199 2286 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33878b480d3344d4ad2b83be036bd4cd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "417a2b170255454e947d7966ee192b21" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 34487 } } peers { permanent_uuid: "33878b480d3344d4ad2b83be036bd4cd" member_type: VOTER last_known_addr { host: "127.20.178.125" port: 35687 } } peers { permanent_uuid: "9628e08b2c2c484bba6eecf4387c7ce6" member_type: VOTER last_known_addr { host: "127.20.178.124" port: 44107 } } }
I20260929 03:20:01.511391 2286 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33878b480d3344d4ad2b83be036bd4cd [sys.catalog]: This master's current role is: FOLLOWER
W20260929 03:20:01.512969 2297 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 33878b480d3344d4ad2b83be036bd4cd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 03:20:01.513067 2297 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 03:20:01.514945 2101 master_runner.cc:263] No leader master found from master 33878b480d3344d4ad2b83be036bd4cd
I20260929 03:20:01.514989 2101 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 03:20:01.529433 2268 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9628e08b2c2c484bba6eecf4387c7ce6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:20:01.529572 2268 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9628e08b2c2c484bba6eecf4387c7ce6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "417a2b170255454e947d7966ee192b21" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 34487 } } peers { permanent_uuid: "33878b480d3344d4ad2b83be036bd4cd" member_type: VOTER last_known_addr { host: "127.20.178.125" port: 35687 } } peers { permanent_uuid: "9628e08b2c2c484bba6eecf4387c7ce6" member_type: VOTER last_known_addr { host: "127.20.178.124" port: 44107 } }
I20260929 03:20:01.529801 2268 leader_election.cc:288] T 00000000000000000000000000000000 P 9628e08b2c2c484bba6eecf4387c7ce6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 417a2b170255454e947d7966ee192b21 (127.20.178.126:34487), 33878b480d3344d4ad2b83be036bd4cd (127.20.178.125:35687)
I20260929 03:20:01.530089 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9628e08b2c2c484bba6eecf4387c7ce6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "417a2b170255454e947d7966ee192b21" is_pre_election: true
I20260929 03:20:01.530243 2068 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 417a2b170255454e947d7966ee192b21 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9628e08b2c2c484bba6eecf4387c7ce6 in term 0.
I20260929 03:20:01.530431 2194 leader_election.cc:302] T 00000000000000000000000000000000 P 9628e08b2c2c484bba6eecf4387c7ce6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 417a2b170255454e947d7966ee192b21, 9628e08b2c2c484bba6eecf4387c7ce6; no voters:
I20260929 03:20:01.530761 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9628e08b2c2c484bba6eecf4387c7ce6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33878b480d3344d4ad2b83be036bd4cd" is_pre_election: true
I20260929 03:20:01.530902 2143 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 33878b480d3344d4ad2b83be036bd4cd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9628e08b2c2c484bba6eecf4387c7ce6 in term 0.
I20260929 03:20:01.531100 2268 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9628e08b2c2c484bba6eecf4387c7ce6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 03:20:01.531162 2268 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9628e08b2c2c484bba6eecf4387c7ce6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 03:20:01.531193 2268 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9628e08b2c2c484bba6eecf4387c7ce6 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:20:01.532083 2268 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9628e08b2c2c484bba6eecf4387c7ce6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "417a2b170255454e947d7966ee192b21" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 34487 } } peers { permanent_uuid: "33878b480d3344d4ad2b83be036bd4cd" member_type: VOTER last_known_addr { host: "127.20.178.125" port: 35687 } } peers { permanent_uuid: "9628e08b2c2c484bba6eecf4387c7ce6" member_type: VOTER last_known_addr { host: "127.20.178.124" port: 44107 } }
I20260929 03:20:01.532207 2268 leader_election.cc:288] T 00000000000000000000000000000000 P 9628e08b2c2c484bba6eecf4387c7ce6 [CANDIDATE]: Term 1 election: Requested vote from peers 417a2b170255454e947d7966ee192b21 (127.20.178.126:34487), 33878b480d3344d4ad2b83be036bd4cd (127.20.178.125:35687)
I20260929 03:20:01.532415 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9628e08b2c2c484bba6eecf4387c7ce6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "417a2b170255454e947d7966ee192b21"
I20260929 03:20:01.532490 2068 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 417a2b170255454e947d7966ee192b21 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:20:01.533075 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9628e08b2c2c484bba6eecf4387c7ce6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33878b480d3344d4ad2b83be036bd4cd"
I20260929 03:20:01.533164 2143 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 33878b480d3344d4ad2b83be036bd4cd [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:20:01.534075 2143 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 33878b480d3344d4ad2b83be036bd4cd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9628e08b2c2c484bba6eecf4387c7ce6 in term 1.
I20260929 03:20:01.534845 2068 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 417a2b170255454e947d7966ee192b21 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9628e08b2c2c484bba6eecf4387c7ce6 in term 1.
I20260929 03:20:01.535019 2194 leader_election.cc:302] T 00000000000000000000000000000000 P 9628e08b2c2c484bba6eecf4387c7ce6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 417a2b170255454e947d7966ee192b21, 9628e08b2c2c484bba6eecf4387c7ce6; no voters:
I20260929 03:20:01.535199 2268 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9628e08b2c2c484bba6eecf4387c7ce6 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:20:01.535389 2268 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9628e08b2c2c484bba6eecf4387c7ce6 [term 1 LEADER]: Becoming Leader. State: Replica: 9628e08b2c2c484bba6eecf4387c7ce6, State: Running, Role: LEADER
I20260929 03:20:01.535485 2268 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9628e08b2c2c484bba6eecf4387c7ce6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "417a2b170255454e947d7966ee192b21" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 34487 } } peers { permanent_uuid: "33878b480d3344d4ad2b83be036bd4cd" member_type: VOTER last_known_addr { host: "127.20.178.125" port: 35687 } } peers { permanent_uuid: "9628e08b2c2c484bba6eecf4387c7ce6" member_type: VOTER last_known_addr { host: "127.20.178.124" port: 44107 } }
I20260929 03:20:01.535900 2268 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9628e08b2c2c484bba6eecf4387c7ce6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9628e08b2c2c484bba6eecf4387c7ce6. Latest consensus state: current_term: 1 leader_uuid: "9628e08b2c2c484bba6eecf4387c7ce6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "417a2b170255454e947d7966ee192b21" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 34487 } } peers { permanent_uuid: "33878b480d3344d4ad2b83be036bd4cd" member_type: VOTER last_known_addr { host: "127.20.178.125" port: 35687 } } peers { permanent_uuid: "9628e08b2c2c484bba6eecf4387c7ce6" member_type: VOTER last_known_addr { host: "127.20.178.124" port: 44107 } } }
I20260929 03:20:01.535981 2268 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9628e08b2c2c484bba6eecf4387c7ce6 [sys.catalog]: This master's current role is: LEADER
I20260929 03:20:01.536211 2301 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:20:01.536410 2301 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:20:01.539505 2143 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 33878b480d3344d4ad2b83be036bd4cd [term 1 FOLLOWER]: Refusing update from remote peer 9628e08b2c2c484bba6eecf4387c7ce6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:20:01.539775 2068 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 417a2b170255454e947d7966ee192b21 [term 1 FOLLOWER]: Refusing update from remote peer 9628e08b2c2c484bba6eecf4387c7ce6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:20:01.540120 2268 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9628e08b2c2c484bba6eecf4387c7ce6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "417a2b170255454e947d7966ee192b21" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 34487 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:20:01.540268 2268 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9628e08b2c2c484bba6eecf4387c7ce6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "33878b480d3344d4ad2b83be036bd4cd" member_type: VOTER last_known_addr { host: "127.20.178.125" port: 35687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:20:01.541697 2286 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33878b480d3344d4ad2b83be036bd4cd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9628e08b2c2c484bba6eecf4387c7ce6. Latest consensus state: current_term: 1 leader_uuid: "9628e08b2c2c484bba6eecf4387c7ce6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "417a2b170255454e947d7966ee192b21" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 34487 } } peers { permanent_uuid: "33878b480d3344d4ad2b83be036bd4cd" member_type: VOTER last_known_addr { host: "127.20.178.125" port: 35687 } } peers { permanent_uuid: "9628e08b2c2c484bba6eecf4387c7ce6" member_type: VOTER last_known_addr { host: "127.20.178.124" port: 44107 } } }
I20260929 03:20:01.541803 2286 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33878b480d3344d4ad2b83be036bd4cd [sys.catalog]: This master's current role is: FOLLOWER
I20260929 03:20:01.542699 2251 sys_catalog.cc:455] T 00000000000000000000000000000000 P 417a2b170255454e947d7966ee192b21 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9628e08b2c2c484bba6eecf4387c7ce6. Latest consensus state: current_term: 1 leader_uuid: "9628e08b2c2c484bba6eecf4387c7ce6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "417a2b170255454e947d7966ee192b21" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 34487 } } peers { permanent_uuid: "33878b480d3344d4ad2b83be036bd4cd" member_type: VOTER last_known_addr { host: "127.20.178.125" port: 35687 } } peers { permanent_uuid: "9628e08b2c2c484bba6eecf4387c7ce6" member_type: VOTER last_known_addr { host: "127.20.178.124" port: 44107 } } }
I20260929 03:20:01.542794 2251 sys_catalog.cc:458] T 00000000000000000000000000000000 P 417a2b170255454e947d7966ee192b21 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 03:20:01.544807 2268 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9628e08b2c2c484bba6eecf4387c7ce6 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9628e08b2c2c484bba6eecf4387c7ce6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "417a2b170255454e947d7966ee192b21" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 34487 } } peers { permanent_uuid: "33878b480d3344d4ad2b83be036bd4cd" member_type: VOTER last_known_addr { host: "127.20.178.125" port: 35687 } } peers { permanent_uuid: "9628e08b2c2c484bba6eecf4387c7ce6" member_type: VOTER last_known_addr { host: "127.20.178.124" port: 44107 } } }
I20260929 03:20:01.544899 2268 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9628e08b2c2c484bba6eecf4387c7ce6 [sys.catalog]: This master's current role is: LEADER
I20260929 03:20:01.545363 2251 sys_catalog.cc:455] T 00000000000000000000000000000000 P 417a2b170255454e947d7966ee192b21 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9628e08b2c2c484bba6eecf4387c7ce6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "417a2b170255454e947d7966ee192b21" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 34487 } } peers { permanent_uuid: "33878b480d3344d4ad2b83be036bd4cd" member_type: VOTER last_known_addr { host: "127.20.178.125" port: 35687 } } peers { permanent_uuid: "9628e08b2c2c484bba6eecf4387c7ce6" member_type: VOTER last_known_addr { host: "127.20.178.124" port: 44107 } } }
I20260929 03:20:01.545455 2251 sys_catalog.cc:458] T 00000000000000000000000000000000 P 417a2b170255454e947d7966ee192b21 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 03:20:01.545672 2302 mvcc.cc:205] Tried to move back new op lower bound from 7334510598303191040 to 7334510598290026496. Current Snapshot: MvccSnapshot[applied={T|T < 7334510598303191040}]
I20260929 03:20:01.545837 2305 mvcc.cc:205] Tried to move back new op lower bound from 7334510598303191040 to 7334510598290026496. Current Snapshot: MvccSnapshot[applied={T|T < 7334510598303191040}]
I20260929 03:20:01.547274 2268 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9628e08b2c2c484bba6eecf4387c7ce6 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9628e08b2c2c484bba6eecf4387c7ce6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "417a2b170255454e947d7966ee192b21" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 34487 } } peers { permanent_uuid: "33878b480d3344d4ad2b83be036bd4cd" member_type: VOTER last_known_addr { host: "127.20.178.125" port: 35687 } } peers { permanent_uuid: "9628e08b2c2c484bba6eecf4387c7ce6" member_type: VOTER last_known_addr { host: "127.20.178.124" port: 44107 } } }
I20260929 03:20:01.547359 2268 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9628e08b2c2c484bba6eecf4387c7ce6 [sys.catalog]: This master's current role is: LEADER
I20260929 03:20:01.547665 2286 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33878b480d3344d4ad2b83be036bd4cd [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9628e08b2c2c484bba6eecf4387c7ce6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "417a2b170255454e947d7966ee192b21" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 34487 } } peers { permanent_uuid: "33878b480d3344d4ad2b83be036bd4cd" member_type: VOTER last_known_addr { host: "127.20.178.125" port: 35687 } } peers { permanent_uuid: "9628e08b2c2c484bba6eecf4387c7ce6" member_type: VOTER last_known_addr { host: "127.20.178.124" port: 44107 } } }
I20260929 03:20:01.547749 2286 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33878b480d3344d4ad2b83be036bd4cd [sys.catalog]: This master's current role is: FOLLOWER
I20260929 03:20:01.548115 2303 mvcc.cc:205] Tried to move back new op lower bound from 7334510598303191040 to 7334510598290026496. Current Snapshot: MvccSnapshot[applied={T|T < 7334510598303191040}]
I20260929 03:20:01.548991 2301 catalog_manager.cc:1347] Generated new cluster ID: e6370f96b6b64ae88f6a0507e21780e5
I20260929 03:20:01.549047 2301 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:20:01.564934 2301 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:20:01.565676 2301 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:20:01.578284 2301 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9628e08b2c2c484bba6eecf4387c7ce6: Generated new TSK 0
I20260929 03:20:01.578500 2301 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 03:20:02.461289 2262 catalog_manager.cc:1259] Loaded cluster ID: e6370f96b6b64ae88f6a0507e21780e5
I20260929 03:20:02.461369 2262 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 417a2b170255454e947d7966ee192b21: loading cluster ID for follower catalog manager: success
I20260929 03:20:02.462200 2262 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 417a2b170255454e947d7966ee192b21: acquiring CA information for follower catalog manager: success
I20260929 03:20:02.462527 2262 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 417a2b170255454e947d7966ee192b21: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 03:20:02.513520 2297 catalog_manager.cc:1259] Loaded cluster ID: e6370f96b6b64ae88f6a0507e21780e5
I20260929 03:20:02.513597 2297 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 33878b480d3344d4ad2b83be036bd4cd: loading cluster ID for follower catalog manager: success
I20260929 03:20:02.514446 2297 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 33878b480d3344d4ad2b83be036bd4cd: acquiring CA information for follower catalog manager: success
I20260929 03:20:02.514765 2297 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 33878b480d3344d4ad2b83be036bd4cd: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 03:20:02.637588 21193 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuoaHbX/build/release/bin/kudu with pid 2026
I20260929 03:20:02.642694 21193 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuoaHbX/build/release/bin/kudu with pid 2101
I20260929 03:20:02.647524 21193 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuoaHbX/build/release/bin/kudu with pid 2176
2026-09-29T03:20:02Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (2821 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790651852476208-21193-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:20:02 dist-test-slave-c9qh krb5kdc[2317](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:20:02 dist-test-slave-c9qh krb5kdc[2317](info): set up 2 sockets
Sep 29 03:20:02 dist-test-slave-c9qh krb5kdc[2317](info): commencing operation
krb5kdc: starting...
W20260929 03:20:03.672758 21193 mini_kdc.cc:121] Time spent starting KDC: real 1.005s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790651852476208-21193-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790651852476208-21193-0/krb5kdc/test-user.keytab.
Sep 29 03:20:03 dist-test-slave-c9qh krb5kdc[2317](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652003, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:20:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:20:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.178.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.178.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.178.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.126.keytab.
Entry for principal kudu/127.20.178.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.126.keytab.
I20260929 03:20:03.792305 21193 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuoaHbX/build/release/bin/kudu
/tmp/dist-test-taskuoaHbX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790651852476208-21193-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790651852476208-21193-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790651852476208-21193-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790651852476208-21193-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.178.126:44853
--webserver_interface=127.20.178.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.126.keytab
--principal=kudu/127.20.178.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.178.84:42385
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.178.126:44853
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790651852476208-21193-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790651852476208-21193-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790651852476208-21193-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:20:03.873325 2330 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:20:03.873576 2330 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:20:03.873612 2330 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:20:03.874770 2330 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:20:03.874855 2330 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:20:03.874881 2330 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:20:03.874923 2330 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:20:03.874954 2330 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:20:03.876444 2330 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.178.84:42385
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790651852476208-21193-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790651852476208-21193-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.178.126:44853
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.126.keytab
--principal=kudu/127.20.178.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.178.126:44853
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790651852476208-21193-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.178.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2330
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790651852476208-21193-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:20:03.876744 2330 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:20:03.876981 2330 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:20:03.881843 2336 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:20:03.881840 2338 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:20:03.881927 2330 server_base.cc:1034] running on GCE node
W20260929 03:20:03.881839 2335 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:20:03.882301 2330 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:20:03.882627 2330 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:20:03.883805 2330 hybrid_clock.cc:648] HybridClock initialized: now 1790652003883797 us; error 40 us; skew 500 ppm
Sep 29 03:20:03 dist-test-slave-c9qh krb5kdc[2317](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652003, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:20:03.886736 2330 init.cc:377] Logged in from keytab as kudu/127.20.178.126@KRBTEST.COM (short username kudu)
I20260929 03:20:03.887207 2330 webserver.cc:460] Webserver started at http://127.20.178.126:43191/ using document root <none> and password file <none>
I20260929 03:20:03.887461 2330 fs_manager.cc:359] Metadata directory not provided
I20260929 03:20:03.887508 2330 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:20:03.887681 2330 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:20:03.888849 2330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790651852476208-21193-0/minicluster-data/master-0/data/instance:
uuid: "9bcccd761e494b0399e3d52222a59920"
format_stamp: "Formatted at 2026-09-29 03:20:03 on dist-test-slave-c9qh"
server_key: "02aeb434b0f0e3c199a3e01ad40931d0"
server_key_iv: "b0b6d334334a8bb2205a7a3af9a4afd9"
server_key_version: "encryptionkey@0"
I20260929 03:20:03.889213 2330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790651852476208-21193-0/minicluster-data/master-0/wal/instance:
uuid: "9bcccd761e494b0399e3d52222a59920"
format_stamp: "Formatted at 2026-09-29 03:20:03 on dist-test-slave-c9qh"
server_key: "02aeb434b0f0e3c199a3e01ad40931d0"
server_key_iv: "b0b6d334334a8bb2205a7a3af9a4afd9"
server_key_version: "encryptionkey@0"
I20260929 03:20:03.890501 2330 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 03:20:03.891254 2345 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:20:03.891433 2330 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 03:20:03.891532 2330 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790651852476208-21193-0/minicluster-data/master-0/data,/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790651852476208-21193-0/minicluster-data/master-0/wal
uuid: "9bcccd761e494b0399e3d52222a59920"
format_stamp: "Formatted at 2026-09-29 03:20:03 on dist-test-slave-c9qh"
server_key: "02aeb434b0f0e3c199a3e01ad40931d0"
server_key_iv: "b0b6d334334a8bb2205a7a3af9a4afd9"
server_key_version: "encryptionkey@0"
I20260929 03:20:03.891624 2330 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790651852476208-21193-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790651852476208-21193-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790651852476208-21193-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:20:03.904171 2330 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:20:03.906121 2330 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:20:03.906297 2330 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:20:03.910324 2330 rpc_server.cc:307] RPC server started. Bound to: 127.20.178.126:44853
I20260929 03:20:03.910386 2397 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.178.126:44853 every 8 connection(s)
I20260929 03:20:03.910743 2330 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790651852476208-21193-0/minicluster-data/master-0/data/info.pb
I20260929 03:20:03.911605 2398 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:20:03.913892 2398 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9bcccd761e494b0399e3d52222a59920: Bootstrap starting.
I20260929 03:20:03.914413 2398 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9bcccd761e494b0399e3d52222a59920: Neither blocks nor log segments found. Creating new log.
I20260929 03:20:03.914652 2398 log.cc:824] T 00000000000000000000000000000000 P 9bcccd761e494b0399e3d52222a59920: Log is configured to *not* fsync() on all Append() calls
I20260929 03:20:03.915268 2398 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9bcccd761e494b0399e3d52222a59920: No bootstrap required, opened a new log
I20260929 03:20:03.916953 21193 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuoaHbX/build/release/bin/kudu as pid 2330
I20260929 03:20:03.917049 21193 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790651852476208-21193-0/minicluster-data/master-0/wal/instance
I20260929 03:20:03.916921 2398 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9bcccd761e494b0399e3d52222a59920 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bcccd761e494b0399e3d52222a59920" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 44853 } }
I20260929 03:20:03.917094 2398 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9bcccd761e494b0399e3d52222a59920 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:20:03.917130 2398 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9bcccd761e494b0399e3d52222a59920 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bcccd761e494b0399e3d52222a59920, State: Initialized, Role: FOLLOWER
I20260929 03:20:03.917239 21193 external_mini_cluster.cc:1428] Setting key 28849e1e9adac9ebb389ca30fe231bfa
I20260929 03:20:03.917284 2398 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9bcccd761e494b0399e3d52222a59920 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bcccd761e494b0399e3d52222a59920" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 44853 } }
I20260929 03:20:03.917356 2398 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9bcccd761e494b0399e3d52222a59920 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:20:03.917438 2398 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9bcccd761e494b0399e3d52222a59920 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:20:03.917524 2398 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9bcccd761e494b0399e3d52222a59920 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:20:03.918416 2398 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9bcccd761e494b0399e3d52222a59920 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bcccd761e494b0399e3d52222a59920" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 44853 } }
I20260929 03:20:03.918503 2398 leader_election.cc:302] T 00000000000000000000000000000000 P 9bcccd761e494b0399e3d52222a59920 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9bcccd761e494b0399e3d52222a59920; no voters:
I20260929 03:20:03.918680 2398 leader_election.cc:288] T 00000000000000000000000000000000 P 9bcccd761e494b0399e3d52222a59920 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:20:03.918715 2403 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9bcccd761e494b0399e3d52222a59920 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:20:03.918848 2403 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9bcccd761e494b0399e3d52222a59920 [term 1 LEADER]: Becoming Leader. State: Replica: 9bcccd761e494b0399e3d52222a59920, State: Running, Role: LEADER
I20260929 03:20:03.918964 2398 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9bcccd761e494b0399e3d52222a59920 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:20:03.919016 2403 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9bcccd761e494b0399e3d52222a59920 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bcccd761e494b0399e3d52222a59920" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 44853 } }
I20260929 03:20:03.919404 2404 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9bcccd761e494b0399e3d52222a59920 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9bcccd761e494b0399e3d52222a59920" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bcccd761e494b0399e3d52222a59920" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 44853 } } }
I20260929 03:20:03.919478 2404 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9bcccd761e494b0399e3d52222a59920 [sys.catalog]: This master's current role is: LEADER
I20260929 03:20:03.919613 2405 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9bcccd761e494b0399e3d52222a59920 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9bcccd761e494b0399e3d52222a59920. Latest consensus state: current_term: 1 leader_uuid: "9bcccd761e494b0399e3d52222a59920" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bcccd761e494b0399e3d52222a59920" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 44853 } } }
I20260929 03:20:03.919658 2405 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9bcccd761e494b0399e3d52222a59920 [sys.catalog]: This master's current role is: LEADER
I20260929 03:20:03.920126 2409 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:20:03.920626 2409 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:20:03.922739 2409 catalog_manager.cc:1347] Generated new cluster ID: 75e6b6b0c10a4145866e074907f63b25
I20260929 03:20:03.922797 2409 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 29 03:20:03 dist-test-slave-c9qh krb5kdc[2317](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652003, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.178.126@KRBTEST.COM
I20260929 03:20:03.927786 2402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:20:03.917666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47234 (local address 127.20.178.126:44853)
0929 03:20:03.917840 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0929 03:20:03.917849 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 03:20:03.917886 (+ 37us) server_negotiation.cc:407] Connection header received
0929 03:20:03.918027 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:20:03.918032 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:20:03.918074 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:20:03.918209 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:20:03.918997 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:20:03.920369 (+ 1372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:20:03.921470 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:20:03.922050 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:20:03.922164 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:20:03.924476 (+ 2312us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:20:03.924484 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:20:03.924484 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:20:03.924494 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:20:03.926245 (+ 1751us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:20:03.926620 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:20:03.926623 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:20:03.926623 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:20:03.926719 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:20:03.926946 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:20:03.926947 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:20:03.926947 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:20:03.927175 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:20:03.927337 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0929 03:20:03.927455 (+ 118us) server_negotiation.cc:299] Negotiation successful
0929 03:20:03.927596 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":89,"threads_started":1}
I20260929 03:20:03.928721 21193 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 03:20:03.932246 2409 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:20:03.932808 2409 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:20:03.941793 2409 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9bcccd761e494b0399e3d52222a59920: Generated new TSK 0
I20260929 03:20:03.942013 2409 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 03:20:03.982561 21193 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuoaHbX/build/release/bin/kudu with pid 2330
2026-09-29T03:20:03Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1333 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:20:04 dist-test-slave-c9qh krb5kdc[2425](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:20:04 dist-test-slave-c9qh krb5kdc[2425](info): set up 2 sockets
Sep 29 03:20:04 dist-test-slave-c9qh krb5kdc[2425](info): commencing operation
krb5kdc: starting...
W20260929 03:20:05.020844 21193 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/krb5kdc/test-user.keytab.
Sep 29 03:20:05 dist-test-slave-c9qh krb5kdc[2425](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652005, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:20:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:20:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.178.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.178.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.178.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.126.keytab.
Entry for principal kudu/127.20.178.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.126.keytab.
I20260929 03:20:05.137298 21193 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuoaHbX/build/release/bin/kudu
/tmp/dist-test-taskuoaHbX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.178.126:33493
--webserver_interface=127.20.178.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.126.keytab
--principal=kudu/127.20.178.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.178.84:42811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.178.126:33493
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:20:05.216506 2438 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:20:05.216728 2438 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:20:05.216751 2438 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:20:05.217845 2438 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:20:05.217880 2438 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:20:05.217893 2438 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:20:05.217906 2438 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:20:05.217918 2438 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:20:05.219307 2438 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.178.84:42811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.178.126:33493
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.126.keytab
--principal=kudu/127.20.178.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.178.126:33493
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.178.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2438
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:20:05.219496 2438 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:20:05.219661 2438 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:20:05.224352 2443 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:20:05.224522 2438 server_base.cc:1034] running on GCE node
W20260929 03:20:05.224344 2446 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:20:05.224332 2444 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:20:05.224879 2438 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:20:05.225188 2438 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:20:05.226353 2438 hybrid_clock.cc:648] HybridClock initialized: now 1790652005226330 us; error 52 us; skew 500 ppm
Sep 29 03:20:05 dist-test-slave-c9qh krb5kdc[2425](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652005, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:20:05.228699 2438 init.cc:377] Logged in from keytab as kudu/127.20.178.126@KRBTEST.COM (short username kudu)
I20260929 03:20:05.229174 2438 webserver.cc:460] Webserver started at http://127.20.178.126:34937/ using document root <none> and password file <none>
I20260929 03:20:05.229416 2438 fs_manager.cc:359] Metadata directory not provided
I20260929 03:20:05.229493 2438 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:20:05.229637 2438 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:20:05.230654 2438 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/minicluster-data/master-0/data/instance:
uuid: "96a8981e1d594b7eaae3ccca9baa3d3e"
format_stamp: "Formatted at 2026-09-29 03:20:05 on dist-test-slave-c9qh"
server_key: "450b0b8b1ce02ef39fd2c655820f5287"
server_key_iv: "0b98de62dc721901f87586f3ae0b23e2"
server_key_version: "encryptionkey@0"
I20260929 03:20:05.231006 2438 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/minicluster-data/master-0/wal/instance:
uuid: "96a8981e1d594b7eaae3ccca9baa3d3e"
format_stamp: "Formatted at 2026-09-29 03:20:05 on dist-test-slave-c9qh"
server_key: "450b0b8b1ce02ef39fd2c655820f5287"
server_key_iv: "0b98de62dc721901f87586f3ae0b23e2"
server_key_version: "encryptionkey@0"
I20260929 03:20:05.232337 2438 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260929 03:20:05.233150 2453 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:20:05.233340 2438 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260929 03:20:05.233446 2438 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/minicluster-data/master-0/data,/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/minicluster-data/master-0/wal
uuid: "96a8981e1d594b7eaae3ccca9baa3d3e"
format_stamp: "Formatted at 2026-09-29 03:20:05 on dist-test-slave-c9qh"
server_key: "450b0b8b1ce02ef39fd2c655820f5287"
server_key_iv: "0b98de62dc721901f87586f3ae0b23e2"
server_key_version: "encryptionkey@0"
I20260929 03:20:05.233518 2438 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:20:05.261281 2438 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:20:05.263306 2438 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:20:05.263478 2438 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:20:05.267705 2438 rpc_server.cc:307] RPC server started. Bound to: 127.20.178.126:33493
I20260929 03:20:05.267742 2505 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.178.126:33493 every 8 connection(s)
I20260929 03:20:05.268124 2438 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/minicluster-data/master-0/data/info.pb
I20260929 03:20:05.269310 2506 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:20:05.271631 2506 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 96a8981e1d594b7eaae3ccca9baa3d3e: Bootstrap starting.
I20260929 03:20:05.271894 21193 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuoaHbX/build/release/bin/kudu as pid 2438
I20260929 03:20:05.272001 21193 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/minicluster-data/master-0/wal/instance
I20260929 03:20:05.272212 21193 external_mini_cluster.cc:1428] Setting key 6f2121a136ca04d9b5f8ec7fa82578ad
I20260929 03:20:05.272382 2506 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 96a8981e1d594b7eaae3ccca9baa3d3e: Neither blocks nor log segments found. Creating new log.
I20260929 03:20:05.272776 2506 log.cc:824] T 00000000000000000000000000000000 P 96a8981e1d594b7eaae3ccca9baa3d3e: Log is configured to *not* fsync() on all Append() calls
I20260929 03:20:05.273530 2506 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 96a8981e1d594b7eaae3ccca9baa3d3e: No bootstrap required, opened a new log
I20260929 03:20:05.275751 2506 raft_consensus.cc:348] T 00000000000000000000000000000000 P 96a8981e1d594b7eaae3ccca9baa3d3e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96a8981e1d594b7eaae3ccca9baa3d3e" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 33493 } }
I20260929 03:20:05.275920 2506 raft_consensus.cc:374] T 00000000000000000000000000000000 P 96a8981e1d594b7eaae3ccca9baa3d3e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:20:05.275954 2506 raft_consensus.cc:729] T 00000000000000000000000000000000 P 96a8981e1d594b7eaae3ccca9baa3d3e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96a8981e1d594b7eaae3ccca9baa3d3e, State: Initialized, Role: FOLLOWER
I20260929 03:20:05.276070 2506 consensus_queue.cc:260] T 00000000000000000000000000000000 P 96a8981e1d594b7eaae3ccca9baa3d3e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96a8981e1d594b7eaae3ccca9baa3d3e" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 33493 } }
I20260929 03:20:05.276165 2506 raft_consensus.cc:388] T 00000000000000000000000000000000 P 96a8981e1d594b7eaae3ccca9baa3d3e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:20:05.276193 2506 raft_consensus.cc:482] T 00000000000000000000000000000000 P 96a8981e1d594b7eaae3ccca9baa3d3e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:20:05.276237 2506 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 96a8981e1d594b7eaae3ccca9baa3d3e [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:20:05.276911 2506 raft_consensus.cc:504] T 00000000000000000000000000000000 P 96a8981e1d594b7eaae3ccca9baa3d3e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96a8981e1d594b7eaae3ccca9baa3d3e" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 33493 } }
I20260929 03:20:05.277000 2506 leader_election.cc:302] T 00000000000000000000000000000000 P 96a8981e1d594b7eaae3ccca9baa3d3e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 96a8981e1d594b7eaae3ccca9baa3d3e; no voters:
I20260929 03:20:05.277215 2506 leader_election.cc:288] T 00000000000000000000000000000000 P 96a8981e1d594b7eaae3ccca9baa3d3e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:20:05.277228 2511 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 96a8981e1d594b7eaae3ccca9baa3d3e [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:20:05.277464 2511 raft_consensus.cc:686] T 00000000000000000000000000000000 P 96a8981e1d594b7eaae3ccca9baa3d3e [term 1 LEADER]: Becoming Leader. State: Replica: 96a8981e1d594b7eaae3ccca9baa3d3e, State: Running, Role: LEADER
I20260929 03:20:05.277483 2506 sys_catalog.cc:564] T 00000000000000000000000000000000 P 96a8981e1d594b7eaae3ccca9baa3d3e [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:20:05.277603 2511 consensus_queue.cc:237] T 00000000000000000000000000000000 P 96a8981e1d594b7eaae3ccca9baa3d3e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96a8981e1d594b7eaae3ccca9baa3d3e" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 33493 } }
I20260929 03:20:05.277948 2511 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96a8981e1d594b7eaae3ccca9baa3d3e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 96a8981e1d594b7eaae3ccca9baa3d3e. Latest consensus state: current_term: 1 leader_uuid: "96a8981e1d594b7eaae3ccca9baa3d3e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96a8981e1d594b7eaae3ccca9baa3d3e" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 33493 } } }
I20260929 03:20:05.277966 2513 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96a8981e1d594b7eaae3ccca9baa3d3e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "96a8981e1d594b7eaae3ccca9baa3d3e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96a8981e1d594b7eaae3ccca9baa3d3e" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 33493 } } }
I20260929 03:20:05.278100 2513 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96a8981e1d594b7eaae3ccca9baa3d3e [sys.catalog]: This master's current role is: LEADER
I20260929 03:20:05.278298 2511 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96a8981e1d594b7eaae3ccca9baa3d3e [sys.catalog]: This master's current role is: LEADER
I20260929 03:20:05.278661 2515 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:20:05.279184 2515 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 29 03:20:05 dist-test-slave-c9qh krb5kdc[2425](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652005, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.178.126@KRBTEST.COM
I20260929 03:20:05.281749 2515 catalog_manager.cc:1347] Generated new cluster ID: 7ac3c4c2fd4743c2a77a251dd9b6ecd1
I20260929 03:20:05.281804 2515 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:20:05.283555 2509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:20:05.272784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56746 (local address 127.20.178.126:33493)
0929 03:20:05.272975 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0929 03:20:05.272985 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 03:20:05.273041 (+ 56us) server_negotiation.cc:407] Connection header received
0929 03:20:05.273592 (+ 551us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:20:05.273602 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:20:05.273665 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:20:05.273792 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:20:05.274569 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:20:05.275405 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:20:05.276479 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:20:05.276954 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:20:05.277100 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:20:05.280131 (+ 3031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:20:05.280134 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:20:05.280134 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:20:05.280144 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:20:05.282049 (+ 1905us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:20:05.282419 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:20:05.282422 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:20:05.282423 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:20:05.282525 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:20:05.282735 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:20:05.282736 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:20:05.282736 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:20:05.282984 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:20:05.283158 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0929 03:20:05.283230 (+ 72us) server_negotiation.cc:299] Negotiation successful
0929 03:20:05.283367 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":94,"threads_started":1}
I20260929 03:20:05.308748 2515 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:20:05.309348 2515 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:20:05.312232 2515 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 96a8981e1d594b7eaae3ccca9baa3d3e: Generated new TSK 0
I20260929 03:20:05.312444 2515 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.178.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.178.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.178.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.65.keytab.
Entry for principal kudu/127.20.178.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.65.keytab.
I20260929 03:20:05.343488 21193 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuoaHbX/build/release/bin/kudu
/tmp/dist-test-taskuoaHbX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.178.65:0
--local_ip_for_outbound_sockets=127.20.178.65
--webserver_interface=127.20.178.65
--webserver_port=0
--tserver_master_addrs=127.20.178.126:33493
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.65.keytab
--principal=kudu/127.20.178.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.178.84:42811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:20:05.441717 2532 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:20:05.441924 2532 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:20:05.441949 2532 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:20:05.443053 2532 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:20:05.443087 2532 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:20:05.443114 2532 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.178.65
I20260929 03:20:05.444483 2532 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.178.84:42811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.65.keytab
--principal=kudu/127.20.178.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.178.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.178.65
--webserver_port=0
--tserver_master_addrs=127.20.178.126:33493
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2532
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.178.65
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:20:05.444723 2532 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:20:05.444911 2532 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:20:05.449867 2540 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:20:05.449892 2537 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:20:05.449867 2538 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:20:05.449988 2532 server_base.cc:1034] running on GCE node
I20260929 03:20:05.450347 2532 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:20:05.450587 2532 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:20:05.451773 2532 hybrid_clock.cc:648] HybridClock initialized: now 1790652005451773 us; error 43 us; skew 500 ppm
Sep 29 03:20:05 dist-test-slave-c9qh krb5kdc[2425](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652005, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:20:05.454311 2532 init.cc:377] Logged in from keytab as kudu/127.20.178.65@KRBTEST.COM (short username kudu)
I20260929 03:20:05.454785 2532 webserver.cc:460] Webserver started at http://127.20.178.65:44209/ using document root <none> and password file <none>
I20260929 03:20:05.455016 2532 fs_manager.cc:359] Metadata directory not provided
I20260929 03:20:05.455083 2532 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:20:05.455235 2532 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:20:05.456352 2532 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/minicluster-data/ts-0/data/instance:
uuid: "5071e4d22012499c9e198b9b39515856"
format_stamp: "Formatted at 2026-09-29 03:20:05 on dist-test-slave-c9qh"
server_key: "3161551a5e4bfdf56ab8fb970e6b2257"
server_key_iv: "bcd44d9111c99f5686d8b9585d62fe55"
server_key_version: "encryptionkey@0"
I20260929 03:20:05.456774 2532 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/minicluster-data/ts-0/wal/instance:
uuid: "5071e4d22012499c9e198b9b39515856"
format_stamp: "Formatted at 2026-09-29 03:20:05 on dist-test-slave-c9qh"
server_key: "3161551a5e4bfdf56ab8fb970e6b2257"
server_key_iv: "bcd44d9111c99f5686d8b9585d62fe55"
server_key_version: "encryptionkey@0"
I20260929 03:20:05.458115 2532 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 03:20:05.458947 2547 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:20:05.459131 2532 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 03:20:05.459230 2532 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/minicluster-data/ts-0/data,/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/minicluster-data/ts-0/wal
uuid: "5071e4d22012499c9e198b9b39515856"
format_stamp: "Formatted at 2026-09-29 03:20:05 on dist-test-slave-c9qh"
server_key: "3161551a5e4bfdf56ab8fb970e6b2257"
server_key_iv: "bcd44d9111c99f5686d8b9585d62fe55"
server_key_version: "encryptionkey@0"
I20260929 03:20:05.459318 2532 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:20:05.496987 2532 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:20:05.499202 2532 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:20:05.499364 2532 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:20:05.499622 2532 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:20:05.499954 2532 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:20:05.500012 2532 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:20:05.500075 2532 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:20:05.500120 2532 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:20:05.506181 2532 rpc_server.cc:307] RPC server started. Bound to: 127.20.178.65:44525
I20260929 03:20:05.506222 2660 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.178.65:44525 every 8 connection(s)
I20260929 03:20:05.506539 2532 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/minicluster-data/ts-0/data/info.pb
I20260929 03:20:05.509279 21193 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuoaHbX/build/release/bin/kudu as pid 2532
I20260929 03:20:05.509393 21193 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/minicluster-data/ts-0/wal/instance
I20260929 03:20:05.509613 21193 external_mini_cluster.cc:1428] Setting key 1b4b7f307461d7df4092d1bd2441087d
Sep 29 03:20:05 dist-test-slave-c9qh krb5kdc[2425](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652005, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.65@KRBTEST.COM for kudu/127.20.178.126@KRBTEST.COM
I20260929 03:20:05.516855 2509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:20:05.507225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.65:50025 (local address 127.20.178.126:33493)
0929 03:20:05.507274 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0929 03:20:05.507277 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:20:05.507442 (+ 165us) server_negotiation.cc:407] Connection header received
0929 03:20:05.507974 (+ 532us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:20:05.507975 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:20:05.507986 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:20:05.508056 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:20:05.509161 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:20:05.509825 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:20:05.511013 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:20:05.511225 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:20:05.511300 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:20:05.513567 (+ 2267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:20:05.513569 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:20:05.513570 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:20:05.513578 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:20:05.515248 (+ 1670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:20:05.515784 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:20:05.515785 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:20:05.515785 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:20:05.515840 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:20:05.516131 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:20:05.516132 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:20:05.516132 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:20:05.516241 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:20:05.516296 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0929 03:20:05.516695 (+ 399us) server_negotiation.cc:299] Negotiation successful
0929 03:20:05.516785 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260929 03:20:05.517104 2663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:20:05.507161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:33493 (local address 127.20.178.65:50025)
0929 03:20:05.507318 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0929 03:20:05.507360 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 03:20:05.507866 (+ 506us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:20:05.508128 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:20:05.508131 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:20:05.508790 (+ 659us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:20:05.509034 (+ 244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:20:05.509039 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:20:05.509993 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:20:05.509994 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:20:05.510901 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:20:05.510903 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:20:05.511322 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:20:05.511323 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:20:05.511455 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:20:05.512009 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:20:05.512019 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:20:05.513471 (+ 1452us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:20:05.515342 (+ 1871us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:20:05.515350 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:20:05.515353 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:20:05.515706 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:20:05.515911 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:20:05.515912 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:20:05.515913 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:20:05.516062 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:20:05.516314 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:20:05.516322 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:20:05.516502 (+ 180us) client_negotiation.cc:769] Sending connection context
0929 03:20:05.516623 (+ 121us) client_negotiation.cc:240] Negotiation successful
0929 03:20:05.516830 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":84,"threads_started":1}
I20260929 03:20:05.517706 2661 heartbeater.cc:347] Connected to a master server at 127.20.178.126:33493
I20260929 03:20:05.517813 2661 heartbeater.cc:464] Registering TS with master...
I20260929 03:20:05.518030 2661 heartbeater.cc:511] Master 127.20.178.126:33493 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.20.178.66@KRBTEST.COM; defaulting to no policy
I20260929 03:20:05.518563 2470 ts_manager.cc:194] Registered new tserver with Master: 5071e4d22012499c9e198b9b39515856 (127.20.178.65:44525)
I20260929 03:20:05.519227 2470 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.178.65@KRBTEST.COM'} at 127.20.178.65:50025
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.178.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.178.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.66.keytab.
Entry for principal kudu/127.20.178.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.66.keytab.
I20260929 03:20:05.535962 21193 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuoaHbX/build/release/bin/kudu
/tmp/dist-test-taskuoaHbX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.178.66:0
--local_ip_for_outbound_sockets=127.20.178.66
--webserver_interface=127.20.178.66
--webserver_port=0
--tserver_master_addrs=127.20.178.126:33493
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.66.keytab
--principal=kudu/127.20.178.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.178.84:42811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:20:05.617092 2666 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:20:05.617352 2666 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:20:05.617388 2666 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:20:05.618530 2666 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:20:05.618610 2666 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:20:05.618656 2666 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.178.66
I20260929 03:20:05.620132 2666 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.178.84:42811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.66.keytab
--principal=kudu/127.20.178.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.178.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.178.66
--webserver_port=0
--tserver_master_addrs=127.20.178.126:33493
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2666
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.178.66
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:20:05.620405 2666 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:20:05.620666 2666 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:20:05.625638 2671 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:20:05.625648 2672 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:20:05.625658 2674 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:20:05.625864 2666 server_base.cc:1034] running on GCE node
I20260929 03:20:05.626156 2666 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:20:05.626397 2666 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:20:05.627540 2666 hybrid_clock.cc:648] HybridClock initialized: now 1790652005627521 us; error 38 us; skew 500 ppm
Sep 29 03:20:05 dist-test-slave-c9qh krb5kdc[2425](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652005, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:20:05.630152 2666 init.cc:377] Logged in from keytab as kudu/127.20.178.66@KRBTEST.COM (short username kudu)
I20260929 03:20:05.630599 2666 webserver.cc:460] Webserver started at http://127.20.178.66:36125/ using document root <none> and password file <none>
I20260929 03:20:05.630790 2666 fs_manager.cc:359] Metadata directory not provided
I20260929 03:20:05.630831 2666 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:20:05.630919 2666 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:20:05.631948 2666 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/minicluster-data/ts-1/data/instance:
uuid: "c4d16a71b9b44e9f95766d5ed5ede8b9"
format_stamp: "Formatted at 2026-09-29 03:20:05 on dist-test-slave-c9qh"
server_key: "53dbba98e9bfa97db74cadca73323601"
server_key_iv: "696b9603205ac7a46e23303ea2764a6f"
server_key_version: "encryptionkey@0"
I20260929 03:20:05.632251 2666 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/minicluster-data/ts-1/wal/instance:
uuid: "c4d16a71b9b44e9f95766d5ed5ede8b9"
format_stamp: "Formatted at 2026-09-29 03:20:05 on dist-test-slave-c9qh"
server_key: "53dbba98e9bfa97db74cadca73323601"
server_key_iv: "696b9603205ac7a46e23303ea2764a6f"
server_key_version: "encryptionkey@0"
I20260929 03:20:05.633632 2666 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 03:20:05.634384 2681 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:20:05.634567 2666 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 03:20:05.634657 2666 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/minicluster-data/ts-1/data,/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/minicluster-data/ts-1/wal
uuid: "c4d16a71b9b44e9f95766d5ed5ede8b9"
format_stamp: "Formatted at 2026-09-29 03:20:05 on dist-test-slave-c9qh"
server_key: "53dbba98e9bfa97db74cadca73323601"
server_key_iv: "696b9603205ac7a46e23303ea2764a6f"
server_key_version: "encryptionkey@0"
I20260929 03:20:05.634734 2666 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:20:05.657222 2666 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:20:05.659281 2666 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:20:05.659395 2666 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:20:05.659577 2666 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:20:05.659904 2666 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:20:05.659941 2666 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:20:05.659991 2666 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:20:05.660017 2666 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:20:05.666006 2666 rpc_server.cc:307] RPC server started. Bound to: 127.20.178.66:37709
I20260929 03:20:05.666091 2794 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.178.66:37709 every 8 connection(s)
I20260929 03:20:05.666415 2666 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/minicluster-data/ts-1/data/info.pb
I20260929 03:20:05.671030 21193 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuoaHbX/build/release/bin/kudu as pid 2666
I20260929 03:20:05.671134 21193 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/minicluster-data/ts-1/wal/instance
I20260929 03:20:05.671368 21193 external_mini_cluster.cc:1428] Setting key 79f190b2c39583579d6687e059181c2b
Sep 29 03:20:05 dist-test-slave-c9qh krb5kdc[2425](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652005, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.66@KRBTEST.COM for kudu/127.20.178.126@KRBTEST.COM
I20260929 03:20:05.675971 2509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:20:05.667191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.66:55469 (local address 127.20.178.126:33493)
0929 03:20:05.667252 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0929 03:20:05.667254 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:20:05.667465 (+ 211us) server_negotiation.cc:407] Connection header received
0929 03:20:05.667961 (+ 496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:20:05.667962 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:20:05.667972 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:20:05.668042 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:20:05.669052 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:20:05.669540 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:20:05.670473 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:20:05.670663 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:20:05.670716 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:20:05.673126 (+ 2410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:20:05.673127 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:20:05.673128 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:20:05.673136 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:20:05.674558 (+ 1422us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:20:05.674981 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:20:05.674982 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:20:05.674982 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:20:05.675047 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:20:05.675299 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:20:05.675300 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:20:05.675300 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:20:05.675417 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:20:05.675493 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0929 03:20:05.675789 (+ 296us) server_negotiation.cc:299] Negotiation successful
0929 03:20:05.675880 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260929 03:20:05.676153 2797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:20:05.667164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:33493 (local address 127.20.178.66:55469)
0929 03:20:05.667318 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0929 03:20:05.667361 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 03:20:05.667846 (+ 485us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:20:05.668098 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:20:05.668101 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:20:05.668761 (+ 660us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:20:05.668966 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:20:05.668972 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:20:05.669617 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:20:05.669618 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:20:05.670399 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:20:05.670401 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:20:05.670734 (+ 333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:20:05.670735 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:20:05.670872 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:20:05.671460 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:20:05.671470 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:20:05.673024 (+ 1554us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:20:05.674658 (+ 1634us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:20:05.674666 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:20:05.674668 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:20:05.674896 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:20:05.675113 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:20:05.675114 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:20:05.675114 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:20:05.675230 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:20:05.675485 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:20:05.675492 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:20:05.675642 (+ 150us) client_negotiation.cc:769] Sending connection context
0929 03:20:05.675748 (+ 106us) client_negotiation.cc:240] Negotiation successful
0929 03:20:05.675910 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":85,"threads_started":1}
I20260929 03:20:05.676858 2795 heartbeater.cc:347] Connected to a master server at 127.20.178.126:33493
I20260929 03:20:05.676972 2795 heartbeater.cc:464] Registering TS with master...
I20260929 03:20:05.677170 2795 heartbeater.cc:511] Master 127.20.178.126:33493 requested a full tablet report, sending...
I20260929 03:20:05.677608 2470 ts_manager.cc:194] Registered new tserver with Master: c4d16a71b9b44e9f95766d5ed5ede8b9 (127.20.178.66:37709)
I20260929 03:20:05.678184 2470 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.178.66@KRBTEST.COM'} at 127.20.178.66:55469
WARNING: no policy specified for kudu/127.20.178.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.178.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.178.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.67.keytab.
Entry for principal kudu/127.20.178.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.67.keytab.
I20260929 03:20:05.698947 21193 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuoaHbX/build/release/bin/kudu
/tmp/dist-test-taskuoaHbX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.178.67:0
--local_ip_for_outbound_sockets=127.20.178.67
--webserver_interface=127.20.178.67
--webserver_port=0
--tserver_master_addrs=127.20.178.126:33493
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.67.keytab
--principal=kudu/127.20.178.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.178.84:42811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:20:05.779376 2800 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:20:05.779582 2800 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:20:05.779608 2800 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:20:05.780731 2800 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:20:05.780766 2800 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:20:05.780791 2800 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.178.67
I20260929 03:20:05.782188 2800 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.178.84:42811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.67.keytab
--principal=kudu/127.20.178.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.178.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.178.67
--webserver_port=0
--tserver_master_addrs=127.20.178.126:33493
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2800
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.178.67
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:20:05.782436 2800 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:20:05.782693 2800 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:20:05.789407 2808 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:20:05.789408 2805 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:20:05.789557 2800 server_base.cc:1034] running on GCE node
W20260929 03:20:05.790061 2806 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:20:05.790284 2800 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:20:05.790550 2800 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:20:05.791718 2800 hybrid_clock.cc:648] HybridClock initialized: now 1790652005791695 us; error 38 us; skew 500 ppm
Sep 29 03:20:05 dist-test-slave-c9qh krb5kdc[2425](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652005, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:20:05.794394 2800 init.cc:377] Logged in from keytab as kudu/127.20.178.67@KRBTEST.COM (short username kudu)
I20260929 03:20:05.794780 2800 webserver.cc:460] Webserver started at http://127.20.178.67:34703/ using document root <none> and password file <none>
I20260929 03:20:05.794974 2800 fs_manager.cc:359] Metadata directory not provided
I20260929 03:20:05.795022 2800 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:20:05.795111 2800 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:20:05.796108 2800 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/minicluster-data/ts-2/data/instance:
uuid: "0fdb3aa2f42046dda60d96844a22c272"
format_stamp: "Formatted at 2026-09-29 03:20:05 on dist-test-slave-c9qh"
server_key: "5e809bd46f719d2e6cf2f84c6b87e4d6"
server_key_iv: "2d588bc3d30278eb60f5ddc3fa8b86c0"
server_key_version: "encryptionkey@0"
I20260929 03:20:05.796411 2800 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/minicluster-data/ts-2/wal/instance:
uuid: "0fdb3aa2f42046dda60d96844a22c272"
format_stamp: "Formatted at 2026-09-29 03:20:05 on dist-test-slave-c9qh"
server_key: "5e809bd46f719d2e6cf2f84c6b87e4d6"
server_key_iv: "2d588bc3d30278eb60f5ddc3fa8b86c0"
server_key_version: "encryptionkey@0"
I20260929 03:20:05.797725 2800 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 03:20:05.798429 2815 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:20:05.798607 2800 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260929 03:20:05.798698 2800 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/minicluster-data/ts-2/data,/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/minicluster-data/ts-2/wal
uuid: "0fdb3aa2f42046dda60d96844a22c272"
format_stamp: "Formatted at 2026-09-29 03:20:05 on dist-test-slave-c9qh"
server_key: "5e809bd46f719d2e6cf2f84c6b87e4d6"
server_key_iv: "2d588bc3d30278eb60f5ddc3fa8b86c0"
server_key_version: "encryptionkey@0"
I20260929 03:20:05.798770 2800 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:20:05.812335 2800 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:20:05.814965 2800 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:20:05.815143 2800 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:20:05.815433 2800 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:20:05.815779 2800 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:20:05.815842 2800 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:20:05.815884 2800 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:20:05.815910 2800 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:20:05.823467 2800 rpc_server.cc:307] RPC server started. Bound to: 127.20.178.67:37489
I20260929 03:20:05.823555 2928 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.178.67:37489 every 8 connection(s)
I20260929 03:20:05.823843 2800 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/minicluster-data/ts-2/data/info.pb
I20260929 03:20:05.824097 21193 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuoaHbX/build/release/bin/kudu as pid 2800
I20260929 03:20:05.824184 21193 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/minicluster-data/ts-2/wal/instance
I20260929 03:20:05.824373 21193 external_mini_cluster.cc:1428] Setting key 74aab1fe455bb70446d8d26641adcefc
Sep 29 03:20:05 dist-test-slave-c9qh krb5kdc[2425](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652005, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.67@KRBTEST.COM for kudu/127.20.178.126@KRBTEST.COM
I20260929 03:20:05.833210 2509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:20:05.824730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.67:33165 (local address 127.20.178.126:33493)
0929 03:20:05.824833 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0929 03:20:05.824835 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:20:05.825277 (+ 442us) server_negotiation.cc:407] Connection header received
0929 03:20:05.825845 (+ 568us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:20:05.825846 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:20:05.825859 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:20:05.825928 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:20:05.826897 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:20:05.827385 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:20:05.828335 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:20:05.828511 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:20:05.828565 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:20:05.830693 (+ 2128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:20:05.830694 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:20:05.830694 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:20:05.830701 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:20:05.831972 (+ 1271us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:20:05.832332 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:20:05.832332 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:20:05.832333 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:20:05.832384 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:20:05.832592 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:20:05.832593 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:20:05.832593 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:20:05.832763 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:20:05.832815 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0929 03:20:05.833082 (+ 267us) server_negotiation.cc:299] Negotiation successful
0929 03:20:05.833143 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260929 03:20:05.833381 2931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:20:05.824708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:33493 (local address 127.20.178.67:33165)
0929 03:20:05.824963 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0929 03:20:05.825183 (+ 220us) client_negotiation.cc:174] Beginning negotiation
0929 03:20:05.825735 (+ 552us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:20:05.825982 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:20:05.825985 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:20:05.826619 (+ 634us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:20:05.826812 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:20:05.826817 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:20:05.827463 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:20:05.827464 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:20:05.828262 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:20:05.828264 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:20:05.828586 (+ 322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:20:05.828587 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:20:05.828742 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:20:05.829201 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:20:05.829209 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:20:05.830589 (+ 1380us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:20:05.832048 (+ 1459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:20:05.832053 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:20:05.832055 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:20:05.832267 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:20:05.832441 (+ 174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:20:05.832441 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:20:05.832442 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:20:05.832538 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:20:05.832818 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:20:05.832822 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:20:05.832947 (+ 125us) client_negotiation.cc:769] Sending connection context
0929 03:20:05.833057 (+ 110us) client_negotiation.cc:240] Negotiation successful
0929 03:20:05.833179 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":75,"threads_started":1}
I20260929 03:20:05.833864 2929 heartbeater.cc:347] Connected to a master server at 127.20.178.126:33493
I20260929 03:20:05.833976 2929 heartbeater.cc:464] Registering TS with master...
I20260929 03:20:05.834159 2929 heartbeater.cc:511] Master 127.20.178.126:33493 requested a full tablet report, sending...
I20260929 03:20:05.834545 2470 ts_manager.cc:194] Registered new tserver with Master: 0fdb3aa2f42046dda60d96844a22c272 (127.20.178.67:37489)
I20260929 03:20:05.834918 2470 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.178.67@KRBTEST.COM'} at 127.20.178.67:33165
I20260929 03:20:05.836071 21193 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 03:20:05.842886 2509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:20:05.837449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56756 (local address 127.20.178.126:33493)
0929 03:20:05.837521 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0929 03:20:05.837523 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:20:05.837604 (+ 81us) server_negotiation.cc:407] Connection header received
0929 03:20:05.837653 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:20:05.837654 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:20:05.837662 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:20:05.837758 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:20:05.838270 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:20:05.838727 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:20:05.839432 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:20:05.839619 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:20:05.839674 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:20:05.840801 (+ 1127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:20:05.840802 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:20:05.840802 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:20:05.840808 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:20:05.841932 (+ 1124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:20:05.842257 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:20:05.842257 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:20:05.842258 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:20:05.842307 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:20:05.842451 (+ 144us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:20:05.842451 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:20:05.842451 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:20:05.842586 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:20:05.842655 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0929 03:20:05.842747 (+ 92us) server_negotiation.cc:299] Negotiation successful
0929 03:20:05.842801 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260929 03:20:05.844318 2470 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56756:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 03:20:05.845896 2470 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 03:20:05.852473 2941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:20:05.848158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.66:37709 (local address 127.0.0.1:44568)
0929 03:20:05.848716 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0929 03:20:05.848726 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 03:20:05.848773 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:20:05.849122 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:20:05.849123 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:20:05.849124 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:20:05.849169 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:20:05.849171 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:20:05.850033 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:20:05.850034 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:20:05.851242 (+ 1208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:20:05.851244 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:20:05.852167 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:20:05.852168 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:20:05.852260 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:20:05.852276 (+ 16us) client_negotiation.cc:769] Sending connection context
0929 03:20:05.852341 (+ 65us) client_negotiation.cc:240] Negotiation successful
0929 03:20:05.852394 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":539,"mutex_wait_us":2,"thread_start_us":55,"threads_started":1}
I20260929 03:20:05.852578 2940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:20:05.848158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.67:37489 (local address 127.0.0.1:45842)
0929 03:20:05.848333 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0929 03:20:05.848372 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 03:20:05.848434 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:20:05.848951 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:20:05.848955 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:20:05.848958 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:20:05.849041 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:20:05.849043 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:20:05.850401 (+ 1358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:20:05.850402 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:20:05.851415 (+ 1013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:20:05.851417 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:20:05.852290 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:20:05.852291 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:20:05.852365 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:20:05.852369 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 03:20:05.852449 (+ 80us) client_negotiation.cc:240] Negotiation successful
0929 03:20:05.852496 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":98,"threads_started":1}
I20260929 03:20:05.852885 2943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:20:05.848253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44568 (local address 127.20.178.66:37709)
0929 03:20:05.848604 (+ 351us) server_negotiation.cc:206] Beginning negotiation
0929 03:20:05.848608 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:20:05.848900 (+ 292us) server_negotiation.cc:407] Connection header received
0929 03:20:05.848946 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:20:05.848948 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:20:05.848962 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:20:05.849042 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:20:05.849227 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:20:05.849926 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:20:05.851325 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:20:05.852075 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:20:05.852153 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:20:05.852215 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0929 03:20:05.852745 (+ 530us) server_negotiation.cc:299] Negotiation successful
0929 03:20:05.852792 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315,"thread_start_us":68,"threads_started":1}
I20260929 03:20:05.852944 2944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:20:05.848394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45842 (local address 127.20.178.67:37489)
0929 03:20:05.848705 (+ 311us) server_negotiation.cc:206] Beginning negotiation
0929 03:20:05.848709 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:20:05.848756 (+ 47us) server_negotiation.cc:407] Connection header received
0929 03:20:05.848798 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:20:05.848800 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:20:05.848817 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:20:05.849125 (+ 308us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:20:05.849205 (+ 80us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:20:05.850307 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:20:05.851503 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:20:05.852188 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:20:05.852282 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:20:05.852335 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0929 03:20:05.852466 (+ 131us) server_negotiation.cc:299] Negotiation successful
0929 03:20:05.852514 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":98,"threads_started":1}
I20260929 03:20:05.853426 2942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:20:05.848160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.65:44525 (local address 127.0.0.1:55760)
0929 03:20:05.848592 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0929 03:20:05.848601 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 03:20:05.848729 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:20:05.849030 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:20:05.849031 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:20:05.849033 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:20:05.849086 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:20:05.849087 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:20:05.851483 (+ 2396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:20:05.851484 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:20:05.852174 (+ 690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:20:05.852175 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:20:05.853215 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:20:05.853216 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:20:05.853276 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:20:05.853280 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 03:20:05.853328 (+ 48us) client_negotiation.cc:240] Negotiation successful
0929 03:20:05.853365 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":96,"threads_started":1}
I20260929 03:20:05.853710 2945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:20:05.848407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55760 (local address 127.20.178.65:44525)
0929 03:20:05.848594 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0929 03:20:05.848600 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:20:05.848681 (+ 81us) server_negotiation.cc:407] Connection header received
0929 03:20:05.848843 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:20:05.848846 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:20:05.848862 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:20:05.848920 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:20:05.849201 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:20:05.850287 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:20:05.852264 (+ 1977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:20:05.853056 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:20:05.853121 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:20:05.853178 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0929 03:20:05.853586 (+ 408us) server_negotiation.cc:299] Negotiation successful
0929 03:20:05.853636 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":90,"threads_started":1}
I20260929 03:20:05.853612 2729 tablet_service.cc:1467] Processing CreateTablet for tablet 8b1ff8fa9d954a688969cd4053496009 (DEFAULT_TABLE table=test-table [id=39e856c9a3c24348bedae2bd3330212e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:20:05.853616 2863 tablet_service.cc:1467] Processing CreateTablet for tablet 8b1ff8fa9d954a688969cd4053496009 (DEFAULT_TABLE table=test-table [id=39e856c9a3c24348bedae2bd3330212e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:20:05.853951 2729 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b1ff8fa9d954a688969cd4053496009. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:20:05.853945 2863 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b1ff8fa9d954a688969cd4053496009. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:20:05.854339 2595 tablet_service.cc:1467] Processing CreateTablet for tablet 8b1ff8fa9d954a688969cd4053496009 (DEFAULT_TABLE table=test-table [id=39e856c9a3c24348bedae2bd3330212e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:20:05.854620 2595 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b1ff8fa9d954a688969cd4053496009. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:20:05.857069 2946 tablet_bootstrap.cc:492] T 8b1ff8fa9d954a688969cd4053496009 P c4d16a71b9b44e9f95766d5ed5ede8b9: Bootstrap starting.
I20260929 03:20:05.857453 2947 tablet_bootstrap.cc:492] T 8b1ff8fa9d954a688969cd4053496009 P 0fdb3aa2f42046dda60d96844a22c272: Bootstrap starting.
I20260929 03:20:05.857779 2946 tablet_bootstrap.cc:654] T 8b1ff8fa9d954a688969cd4053496009 P c4d16a71b9b44e9f95766d5ed5ede8b9: Neither blocks nor log segments found. Creating new log.
I20260929 03:20:05.857834 2948 tablet_bootstrap.cc:492] T 8b1ff8fa9d954a688969cd4053496009 P 5071e4d22012499c9e198b9b39515856: Bootstrap starting.
I20260929 03:20:05.858304 2947 tablet_bootstrap.cc:654] T 8b1ff8fa9d954a688969cd4053496009 P 0fdb3aa2f42046dda60d96844a22c272: Neither blocks nor log segments found. Creating new log.
I20260929 03:20:05.858418 2948 tablet_bootstrap.cc:654] T 8b1ff8fa9d954a688969cd4053496009 P 5071e4d22012499c9e198b9b39515856: Neither blocks nor log segments found. Creating new log.
I20260929 03:20:05.858399 2946 log.cc:824] T 8b1ff8fa9d954a688969cd4053496009 P c4d16a71b9b44e9f95766d5ed5ede8b9: Log is configured to *not* fsync() on all Append() calls
I20260929 03:20:05.858825 2947 log.cc:824] T 8b1ff8fa9d954a688969cd4053496009 P 0fdb3aa2f42046dda60d96844a22c272: Log is configured to *not* fsync() on all Append() calls
I20260929 03:20:05.858958 2948 log.cc:824] T 8b1ff8fa9d954a688969cd4053496009 P 5071e4d22012499c9e198b9b39515856: Log is configured to *not* fsync() on all Append() calls
I20260929 03:20:05.859236 2946 tablet_bootstrap.cc:492] T 8b1ff8fa9d954a688969cd4053496009 P c4d16a71b9b44e9f95766d5ed5ede8b9: No bootstrap required, opened a new log
I20260929 03:20:05.859320 2946 ts_tablet_manager.cc:1397] T 8b1ff8fa9d954a688969cd4053496009 P c4d16a71b9b44e9f95766d5ed5ede8b9: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 03:20:05.859669 2948 tablet_bootstrap.cc:492] T 8b1ff8fa9d954a688969cd4053496009 P 5071e4d22012499c9e198b9b39515856: No bootstrap required, opened a new log
I20260929 03:20:05.859673 2947 tablet_bootstrap.cc:492] T 8b1ff8fa9d954a688969cd4053496009 P 0fdb3aa2f42046dda60d96844a22c272: No bootstrap required, opened a new log
I20260929 03:20:05.859727 2947 ts_tablet_manager.cc:1397] T 8b1ff8fa9d954a688969cd4053496009 P 0fdb3aa2f42046dda60d96844a22c272: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 03:20:05.859727 2948 ts_tablet_manager.cc:1397] T 8b1ff8fa9d954a688969cd4053496009 P 5071e4d22012499c9e198b9b39515856: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 03:20:05.861094 2948 raft_consensus.cc:348] T 8b1ff8fa9d954a688969cd4053496009 P 5071e4d22012499c9e198b9b39515856 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fdb3aa2f42046dda60d96844a22c272" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 37489 } } peers { permanent_uuid: "c4d16a71b9b44e9f95766d5ed5ede8b9" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 37709 } } peers { permanent_uuid: "5071e4d22012499c9e198b9b39515856" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 44525 } }
I20260929 03:20:05.861212 2948 raft_consensus.cc:374] T 8b1ff8fa9d954a688969cd4053496009 P 5071e4d22012499c9e198b9b39515856 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:20:05.861231 2948 raft_consensus.cc:729] T 8b1ff8fa9d954a688969cd4053496009 P 5071e4d22012499c9e198b9b39515856 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5071e4d22012499c9e198b9b39515856, State: Initialized, Role: FOLLOWER
I20260929 03:20:05.861346 2948 consensus_queue.cc:260] T 8b1ff8fa9d954a688969cd4053496009 P 5071e4d22012499c9e198b9b39515856 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fdb3aa2f42046dda60d96844a22c272" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 37489 } } peers { permanent_uuid: "c4d16a71b9b44e9f95766d5ed5ede8b9" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 37709 } } peers { permanent_uuid: "5071e4d22012499c9e198b9b39515856" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 44525 } }
I20260929 03:20:05.861313 2946 raft_consensus.cc:348] T 8b1ff8fa9d954a688969cd4053496009 P c4d16a71b9b44e9f95766d5ed5ede8b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fdb3aa2f42046dda60d96844a22c272" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 37489 } } peers { permanent_uuid: "c4d16a71b9b44e9f95766d5ed5ede8b9" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 37709 } } peers { permanent_uuid: "5071e4d22012499c9e198b9b39515856" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 44525 } }
I20260929 03:20:05.861479 2946 raft_consensus.cc:374] T 8b1ff8fa9d954a688969cd4053496009 P c4d16a71b9b44e9f95766d5ed5ede8b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:20:05.861510 2946 raft_consensus.cc:729] T 8b1ff8fa9d954a688969cd4053496009 P c4d16a71b9b44e9f95766d5ed5ede8b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4d16a71b9b44e9f95766d5ed5ede8b9, State: Initialized, Role: FOLLOWER
I20260929 03:20:05.861621 2661 heartbeater.cc:503] Master 127.20.178.126:33493 was elected leader, sending a full tablet report...
I20260929 03:20:05.861625 2948 ts_tablet_manager.cc:1428] T 8b1ff8fa9d954a688969cd4053496009 P 5071e4d22012499c9e198b9b39515856: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 03:20:05.861698 2946 consensus_queue.cc:260] T 8b1ff8fa9d954a688969cd4053496009 P c4d16a71b9b44e9f95766d5ed5ede8b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fdb3aa2f42046dda60d96844a22c272" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 37489 } } peers { permanent_uuid: "c4d16a71b9b44e9f95766d5ed5ede8b9" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 37709 } } peers { permanent_uuid: "5071e4d22012499c9e198b9b39515856" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 44525 } }
I20260929 03:20:05.861685 2947 raft_consensus.cc:348] T 8b1ff8fa9d954a688969cd4053496009 P 0fdb3aa2f42046dda60d96844a22c272 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fdb3aa2f42046dda60d96844a22c272" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 37489 } } peers { permanent_uuid: "c4d16a71b9b44e9f95766d5ed5ede8b9" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 37709 } } peers { permanent_uuid: "5071e4d22012499c9e198b9b39515856" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 44525 } }
I20260929 03:20:05.861828 2947 raft_consensus.cc:374] T 8b1ff8fa9d954a688969cd4053496009 P 0fdb3aa2f42046dda60d96844a22c272 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:20:05.861866 2947 raft_consensus.cc:729] T 8b1ff8fa9d954a688969cd4053496009 P 0fdb3aa2f42046dda60d96844a22c272 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fdb3aa2f42046dda60d96844a22c272, State: Initialized, Role: FOLLOWER
I20260929 03:20:05.861968 2947 consensus_queue.cc:260] T 8b1ff8fa9d954a688969cd4053496009 P 0fdb3aa2f42046dda60d96844a22c272 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fdb3aa2f42046dda60d96844a22c272" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 37489 } } peers { permanent_uuid: "c4d16a71b9b44e9f95766d5ed5ede8b9" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 37709 } } peers { permanent_uuid: "5071e4d22012499c9e198b9b39515856" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 44525 } }
I20260929 03:20:05.862051 2795 heartbeater.cc:503] Master 127.20.178.126:33493 was elected leader, sending a full tablet report...
I20260929 03:20:05.862232 2947 ts_tablet_manager.cc:1428] T 8b1ff8fa9d954a688969cd4053496009 P 0fdb3aa2f42046dda60d96844a22c272: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260929 03:20:05.862255 2929 heartbeater.cc:503] Master 127.20.178.126:33493 was elected leader, sending a full tablet report...
I20260929 03:20:05.861996 2946 ts_tablet_manager.cc:1428] T 8b1ff8fa9d954a688969cd4053496009 P c4d16a71b9b44e9f95766d5ed5ede8b9: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
W20260929 03:20:05.917032 2796 tablet.cc:2369] T 8b1ff8fa9d954a688969cd4053496009 P c4d16a71b9b44e9f95766d5ed5ede8b9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 03:20:05.938997 2952 raft_consensus.cc:482] T 8b1ff8fa9d954a688969cd4053496009 P 5071e4d22012499c9e198b9b39515856 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:20:05.939155 2952 raft_consensus.cc:504] T 8b1ff8fa9d954a688969cd4053496009 P 5071e4d22012499c9e198b9b39515856 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fdb3aa2f42046dda60d96844a22c272" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 37489 } } peers { permanent_uuid: "c4d16a71b9b44e9f95766d5ed5ede8b9" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 37709 } } peers { permanent_uuid: "5071e4d22012499c9e198b9b39515856" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 44525 } }
I20260929 03:20:05.939625 2952 leader_election.cc:288] T 8b1ff8fa9d954a688969cd4053496009 P 5071e4d22012499c9e198b9b39515856 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fdb3aa2f42046dda60d96844a22c272 (127.20.178.67:37489), c4d16a71b9b44e9f95766d5ed5ede8b9 (127.20.178.66:37709)
I20260929 03:20:05.942531 2944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:20:05.939888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.65:46955 (local address 127.20.178.67:37489)
0929 03:20:05.939941 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0929 03:20:05.939943 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:20:05.939951 (+ 8us) server_negotiation.cc:407] Connection header received
0929 03:20:05.939991 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:20:05.939992 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:20:05.940002 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:20:05.940057 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:20:05.940238 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:20:05.940742 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:20:05.941770 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:20:05.942157 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:20:05.942241 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:20:05.942251 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0929 03:20:05.942404 (+ 153us) server_negotiation.cc:299] Negotiation successful
0929 03:20:05.942443 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260929 03:20:05.942552 2663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:20:05.939787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.67:37489 (local address 127.20.178.65:46955)
0929 03:20:05.939842 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0929 03:20:05.939854 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 03:20:05.939897 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:20:05.940078 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:20:05.940079 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:20:05.940082 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:20:05.940160 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:20:05.940162 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:20:05.940836 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:20:05.940837 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:20:05.941673 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:20:05.941676 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:20:05.942249 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:20:05.942250 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:20:05.942331 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:20:05.942334 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 03:20:05.942399 (+ 65us) client_negotiation.cc:240] Negotiation successful
0929 03:20:05.942443 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35,"mutex_wait_us":54}
I20260929 03:20:05.942565 2943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:20:05.940041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.65:52961 (local address 127.20.178.66:37709)
0929 03:20:05.940084 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0929 03:20:05.940087 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:20:05.940095 (+ 8us) server_negotiation.cc:407] Connection header received
0929 03:20:05.940121 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:20:05.940122 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:20:05.940132 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:20:05.940208 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:20:05.940368 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:20:05.940896 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:20:05.941824 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:20:05.942207 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:20:05.942296 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:20:05.942304 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0929 03:20:05.942458 (+ 154us) server_negotiation.cc:299] Negotiation successful
0929 03:20:05.942498 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260929 03:20:05.942864 2955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:20:05.939808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.66:37709 (local address 127.20.178.65:52961)
0929 03:20:05.940003 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0929 03:20:05.940013 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 03:20:05.940050 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:20:05.940214 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:20:05.940214 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:20:05.940216 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:20:05.940294 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:20:05.940295 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:20:05.940979 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:20:05.940980 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:20:05.941731 (+ 751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:20:05.941733 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:20:05.942295 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:20:05.942296 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:20:05.942372 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:20:05.942375 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 03:20:05.942437 (+ 62us) client_negotiation.cc:240] Negotiation successful
0929 03:20:05.942479 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":105,"threads_started":1}
I20260929 03:20:05.942952 2883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b1ff8fa9d954a688969cd4053496009" candidate_uuid: "5071e4d22012499c9e198b9b39515856" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fdb3aa2f42046dda60d96844a22c272" is_pre_election: true
I20260929 03:20:05.943135 2883 raft_consensus.cc:2454] T 8b1ff8fa9d954a688969cd4053496009 P 0fdb3aa2f42046dda60d96844a22c272 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5071e4d22012499c9e198b9b39515856 in term 0.
I20260929 03:20:05.943217 2749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b1ff8fa9d954a688969cd4053496009" candidate_uuid: "5071e4d22012499c9e198b9b39515856" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4d16a71b9b44e9f95766d5ed5ede8b9" is_pre_election: true
I20260929 03:20:05.943409 2749 raft_consensus.cc:2454] T 8b1ff8fa9d954a688969cd4053496009 P c4d16a71b9b44e9f95766d5ed5ede8b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5071e4d22012499c9e198b9b39515856 in term 0.
I20260929 03:20:05.943437 2549 leader_election.cc:302] T 8b1ff8fa9d954a688969cd4053496009 P 5071e4d22012499c9e198b9b39515856 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fdb3aa2f42046dda60d96844a22c272, 5071e4d22012499c9e198b9b39515856; no voters:
I20260929 03:20:05.943655 2952 raft_consensus.cc:2781] T 8b1ff8fa9d954a688969cd4053496009 P 5071e4d22012499c9e198b9b39515856 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 03:20:05.943701 2952 raft_consensus.cc:482] T 8b1ff8fa9d954a688969cd4053496009 P 5071e4d22012499c9e198b9b39515856 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 03:20:05.943717 2952 raft_consensus.cc:3037] T 8b1ff8fa9d954a688969cd4053496009 P 5071e4d22012499c9e198b9b39515856 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:20:05.944392 2952 raft_consensus.cc:504] T 8b1ff8fa9d954a688969cd4053496009 P 5071e4d22012499c9e198b9b39515856 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fdb3aa2f42046dda60d96844a22c272" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 37489 } } peers { permanent_uuid: "c4d16a71b9b44e9f95766d5ed5ede8b9" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 37709 } } peers { permanent_uuid: "5071e4d22012499c9e198b9b39515856" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 44525 } }
I20260929 03:20:05.944543 2952 leader_election.cc:288] T 8b1ff8fa9d954a688969cd4053496009 P 5071e4d22012499c9e198b9b39515856 [CANDIDATE]: Term 1 election: Requested vote from peers 0fdb3aa2f42046dda60d96844a22c272 (127.20.178.67:37489), c4d16a71b9b44e9f95766d5ed5ede8b9 (127.20.178.66:37709)
I20260929 03:20:05.944708 2883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b1ff8fa9d954a688969cd4053496009" candidate_uuid: "5071e4d22012499c9e198b9b39515856" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fdb3aa2f42046dda60d96844a22c272"
I20260929 03:20:05.944746 2749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b1ff8fa9d954a688969cd4053496009" candidate_uuid: "5071e4d22012499c9e198b9b39515856" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4d16a71b9b44e9f95766d5ed5ede8b9"
I20260929 03:20:05.944809 2749 raft_consensus.cc:3037] T 8b1ff8fa9d954a688969cd4053496009 P c4d16a71b9b44e9f95766d5ed5ede8b9 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:20:05.944799 2883 raft_consensus.cc:3037] T 8b1ff8fa9d954a688969cd4053496009 P 0fdb3aa2f42046dda60d96844a22c272 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:20:05.945724 2749 raft_consensus.cc:2454] T 8b1ff8fa9d954a688969cd4053496009 P c4d16a71b9b44e9f95766d5ed5ede8b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5071e4d22012499c9e198b9b39515856 in term 1.
I20260929 03:20:05.945724 2883 raft_consensus.cc:2454] T 8b1ff8fa9d954a688969cd4053496009 P 0fdb3aa2f42046dda60d96844a22c272 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5071e4d22012499c9e198b9b39515856 in term 1.
I20260929 03:20:05.945888 2549 leader_election.cc:302] T 8b1ff8fa9d954a688969cd4053496009 P 5071e4d22012499c9e198b9b39515856 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fdb3aa2f42046dda60d96844a22c272, 5071e4d22012499c9e198b9b39515856; no voters:
I20260929 03:20:05.946000 2952 raft_consensus.cc:2781] T 8b1ff8fa9d954a688969cd4053496009 P 5071e4d22012499c9e198b9b39515856 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:20:05.946185 2952 raft_consensus.cc:686] T 8b1ff8fa9d954a688969cd4053496009 P 5071e4d22012499c9e198b9b39515856 [term 1 LEADER]: Becoming Leader. State: Replica: 5071e4d22012499c9e198b9b39515856, State: Running, Role: LEADER
I20260929 03:20:05.946302 2952 consensus_queue.cc:237] T 8b1ff8fa9d954a688969cd4053496009 P 5071e4d22012499c9e198b9b39515856 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fdb3aa2f42046dda60d96844a22c272" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 37489 } } peers { permanent_uuid: "c4d16a71b9b44e9f95766d5ed5ede8b9" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 37709 } } peers { permanent_uuid: "5071e4d22012499c9e198b9b39515856" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 44525 } }
I20260929 03:20:05.947189 2470 catalog_manager.cc:5495] T 8b1ff8fa9d954a688969cd4053496009 P 5071e4d22012499c9e198b9b39515856 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5071e4d22012499c9e198b9b39515856 (127.20.178.65). New cstate: current_term: 1 leader_uuid: "5071e4d22012499c9e198b9b39515856" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fdb3aa2f42046dda60d96844a22c272" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 37489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c4d16a71b9b44e9f95766d5ed5ede8b9" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 37709 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5071e4d22012499c9e198b9b39515856" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 44525 } health_report { overall_health: HEALTHY } } }
I20260929 03:20:05.967408 2945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:20:05.965183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55770 (local address 127.20.178.65:44525)
0929 03:20:05.965247 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0929 03:20:05.965249 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:20:05.965257 (+ 8us) server_negotiation.cc:407] Connection header received
0929 03:20:05.965293 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:20:05.965293 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:20:05.965313 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:20:05.965372 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:20:05.965499 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:20:05.965954 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:20:05.966725 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:20:05.966910 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:20:05.966964 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:20:05.967099 (+ 135us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:20:05.967197 (+ 98us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:20:05.967254 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0929 03:20:05.967326 (+ 72us) server_negotiation.cc:299] Negotiation successful
0929 03:20:05.967341 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260929 03:20:05.969147 2749 raft_consensus.cc:1261] T 8b1ff8fa9d954a688969cd4053496009 P c4d16a71b9b44e9f95766d5ed5ede8b9 [term 1 FOLLOWER]: Refusing update from remote peer 5071e4d22012499c9e198b9b39515856: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:20:05.969147 2883 raft_consensus.cc:1261] T 8b1ff8fa9d954a688969cd4053496009 P 0fdb3aa2f42046dda60d96844a22c272 [term 1 FOLLOWER]: Refusing update from remote peer 5071e4d22012499c9e198b9b39515856: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:20:05.969529 2952 consensus_queue.cc:1035] T 8b1ff8fa9d954a688969cd4053496009 P 5071e4d22012499c9e198b9b39515856 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4d16a71b9b44e9f95766d5ed5ede8b9" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 37709 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:20:05.969657 2956 consensus_queue.cc:1035] T 8b1ff8fa9d954a688969cd4053496009 P 5071e4d22012499c9e198b9b39515856 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fdb3aa2f42046dda60d96844a22c272" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:20:05.972055 2960 mvcc.cc:205] Tried to move back new op lower bound from 7334510616446840832 to 7334510616356900864. Current Snapshot: MvccSnapshot[applied={T|T < 7334510616446840832}]
I20260929 03:20:05.972199 2961 mvcc.cc:205] Tried to move back new op lower bound from 7334510616446840832 to 7334510616356900864. Current Snapshot: MvccSnapshot[applied={T|T < 7334510616446840832}]
I20260929 03:20:05.974134 2470 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56756:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:20:05.974229 2470 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56756:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:20:05.975004 2470 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 96a8981e1d594b7eaae3ccca9baa3d3e: Sending DeleteTablet for 3 replicas of tablet 8b1ff8fa9d954a688969cd4053496009
I20260929 03:20:05.975314 2863 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b1ff8fa9d954a688969cd4053496009 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:20:05 UTC) from {username='kudu', principal='kudu/127.20.178.126@KRBTEST.COM'} at 127.0.0.1:45842
I20260929 03:20:05.975338 2729 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b1ff8fa9d954a688969cd4053496009 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:20:05 UTC) from {username='kudu', principal='kudu/127.20.178.126@KRBTEST.COM'} at 127.0.0.1:44568
I20260929 03:20:05.975521 2595 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b1ff8fa9d954a688969cd4053496009 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:20:05 UTC) from {username='kudu', principal='kudu/127.20.178.126@KRBTEST.COM'} at 127.0.0.1:55760
I20260929 03:20:05.975732 2971 tablet_replica.cc:330] T 8b1ff8fa9d954a688969cd4053496009 P 5071e4d22012499c9e198b9b39515856: stopping tablet replica
I20260929 03:20:05.975793 21193 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuoaHbX/build/release/bin/kudu with pid 2532
I20260929 03:20:05.975834 2971 raft_consensus.cc:2229] T 8b1ff8fa9d954a688969cd4053496009 P 5071e4d22012499c9e198b9b39515856 [term 1 LEADER]: Raft consensus shutting down.
I20260929 03:20:05.975955 2971 raft_consensus.cc:2258] T 8b1ff8fa9d954a688969cd4053496009 P 5071e4d22012499c9e198b9b39515856 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:20:05.976790 2969 tablet_replica.cc:330] T 8b1ff8fa9d954a688969cd4053496009 P 0fdb3aa2f42046dda60d96844a22c272: stopping tablet replica
I20260929 03:20:05.976934 2969 raft_consensus.cc:2229] T 8b1ff8fa9d954a688969cd4053496009 P 0fdb3aa2f42046dda60d96844a22c272 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:20:05.977078 2970 tablet_replica.cc:330] T 8b1ff8fa9d954a688969cd4053496009 P c4d16a71b9b44e9f95766d5ed5ede8b9: stopping tablet replica
I20260929 03:20:05.977089 2969 raft_consensus.cc:2258] T 8b1ff8fa9d954a688969cd4053496009 P 0fdb3aa2f42046dda60d96844a22c272 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:20:05.977176 2970 raft_consensus.cc:2229] T 8b1ff8fa9d954a688969cd4053496009 P c4d16a71b9b44e9f95766d5ed5ede8b9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:20:05.977365 2970 raft_consensus.cc:2258] T 8b1ff8fa9d954a688969cd4053496009 P c4d16a71b9b44e9f95766d5ed5ede8b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:20:05.977907 2970 ts_tablet_manager.cc:1905] T 8b1ff8fa9d954a688969cd4053496009 P c4d16a71b9b44e9f95766d5ed5ede8b9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:20:05.978099 2969 ts_tablet_manager.cc:1905] T 8b1ff8fa9d954a688969cd4053496009 P 0fdb3aa2f42046dda60d96844a22c272: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:20:05.979944 2969 ts_tablet_manager.cc:1918] T 8b1ff8fa9d954a688969cd4053496009 P 0fdb3aa2f42046dda60d96844a22c272: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:20:05.980005 2969 log.cc:1196] T 8b1ff8fa9d954a688969cd4053496009 P 0fdb3aa2f42046dda60d96844a22c272: Deleting WAL directory at /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/minicluster-data/ts-2/wal/wals/8b1ff8fa9d954a688969cd4053496009
I20260929 03:20:05.980110 2970 ts_tablet_manager.cc:1918] T 8b1ff8fa9d954a688969cd4053496009 P c4d16a71b9b44e9f95766d5ed5ede8b9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:20:05.980211 2969 ts_tablet_manager.cc:1939] T 8b1ff8fa9d954a688969cd4053496009 P 0fdb3aa2f42046dda60d96844a22c272: Deleting consensus metadata
I20260929 03:20:05.980268 2970 log.cc:1196] T 8b1ff8fa9d954a688969cd4053496009 P c4d16a71b9b44e9f95766d5ed5ede8b9: Deleting WAL directory at /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/minicluster-data/ts-1/wal/wals/8b1ff8fa9d954a688969cd4053496009
I20260929 03:20:05.980551 2455 catalog_manager.cc:4841] TS 0fdb3aa2f42046dda60d96844a22c272 (127.20.178.67:37489): tablet 8b1ff8fa9d954a688969cd4053496009 (table test-table [id=39e856c9a3c24348bedae2bd3330212e]) successfully deleted
I20260929 03:20:05.980595 2970 ts_tablet_manager.cc:1939] T 8b1ff8fa9d954a688969cd4053496009 P c4d16a71b9b44e9f95766d5ed5ede8b9: Deleting consensus metadata
I20260929 03:20:05.981042 2454 catalog_manager.cc:4841] TS c4d16a71b9b44e9f95766d5ed5ede8b9 (127.20.178.66:37709): tablet 8b1ff8fa9d954a688969cd4053496009 (table test-table [id=39e856c9a3c24348bedae2bd3330212e]) successfully deleted
W20260929 03:20:05.981783 2456 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.20.178.65:44525 (error 108)
I20260929 03:20:05.981958 21193 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuoaHbX/build/release/bin/kudu with pid 2666
I20260929 03:20:05.982507 2942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:20:05.982186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.65:44525 (local address 127.0.0.1:55772)
0929 03:20:05.982446 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0929 03:20:05.982476 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.178.65:44525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":243}
W20260929 03:20:05.982733 2456 catalog_manager.cc:4584] TS 5071e4d22012499c9e198b9b39515856 (127.20.178.65:44525): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8b1ff8fa9d954a688969cd4053496009: Network error: Client connection negotiation failed: client connection to 127.20.178.65:44525: connect: Connection refused (error 111)
I20260929 03:20:05.987155 21193 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuoaHbX/build/release/bin/kudu with pid 2800
I20260929 03:20:05.991600 21193 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuoaHbX/build/release/bin/kudu with pid 2438
I20260929 03:20:05.996837 21193 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuoaHbX/build/release/bin/kudu
/tmp/dist-test-taskuoaHbX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.178.126:33493
--webserver_interface=127.20.178.126
--webserver_port=34937
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.126.keytab
--principal=kudu/127.20.178.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.178.84:42811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.178.126:33493
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:20:06.079394 2973 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:20:06.079643 2973 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:20:06.079698 2973 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:20:06.080951 2973 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:20:06.081034 2973 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:20:06.081087 2973 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:20:06.081117 2973 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:20:06.081142 2973 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:20:06.082645 2973 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.178.84:42811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.20.178.126:33493
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.126.keytab
--principal=kudu/127.20.178.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.178.126:33493
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.178.126
--webserver_port=34937
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2973
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:20:06.082921 2973 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:20:06.083156 2973 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 03:20:06.092012 2973 server_base.cc:1034] running on GCE node
W20260929 03:20:06.091792 2982 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:20:06.091871 2979 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:20:06.091792 2980 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:20:06.092499 2973 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:20:06.092881 2973 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:20:06.094040 2973 hybrid_clock.cc:648] HybridClock initialized: now 1790652006094016 us; error 38 us; skew 500 ppm
Sep 29 03:20:06 dist-test-slave-c9qh krb5kdc[2425](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652006, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:20:06.100312 2973 init.cc:377] Logged in from keytab as kudu/127.20.178.126@KRBTEST.COM (short username kudu)
I20260929 03:20:06.100860 2973 webserver.cc:460] Webserver started at http://127.20.178.126:34937/ using document root <none> and password file <none>
I20260929 03:20:06.101120 2973 fs_manager.cc:359] Metadata directory not provided
I20260929 03:20:06.101176 2973 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:20:06.102509 2973 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260929 03:20:06.103139 2989 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:20:06.103328 2973 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 03:20:06.103421 2973 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/minicluster-data/master-0/data,/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/minicluster-data/master-0/wal
uuid: "96a8981e1d594b7eaae3ccca9baa3d3e"
format_stamp: "Formatted at 2026-09-29 03:20:05 on dist-test-slave-c9qh"
server_key: "450b0b8b1ce02ef39fd2c655820f5287"
server_key_iv: "0b98de62dc721901f87586f3ae0b23e2"
server_key_version: "encryptionkey@0"
I20260929 03:20:06.103818 2973 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:20:06.111728 2973 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:20:06.113571 2973 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:20:06.113723 2973 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:20:06.117645 2973 rpc_server.cc:307] RPC server started. Bound to: 127.20.178.126:33493
I20260929 03:20:06.117674 3041 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.178.126:33493 every 8 connection(s)
I20260929 03:20:06.118063 2973 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/minicluster-data/master-0/data/info.pb
I20260929 03:20:06.119623 3042 sys_catalog.cc:263] Verifying existing consensus state
I20260929 03:20:06.120311 3042 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 96a8981e1d594b7eaae3ccca9baa3d3e: Bootstrap starting.
I20260929 03:20:06.121356 21193 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuoaHbX/build/release/bin/kudu as pid 2973
I20260929 03:20:06.121634 21193 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuoaHbX/build/release/bin/kudu
/tmp/dist-test-taskuoaHbX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.178.65:44525
--local_ip_for_outbound_sockets=127.20.178.65
--tserver_master_addrs=127.20.178.126:33493
--webserver_port=44209
--webserver_interface=127.20.178.65
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.65.keytab
--principal=kudu/127.20.178.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.178.84:42811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 03:20:06.128861 3042 log.cc:824] T 00000000000000000000000000000000 P 96a8981e1d594b7eaae3ccca9baa3d3e: Log is configured to *not* fsync() on all Append() calls
I20260929 03:20:06.130831 3042 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 96a8981e1d594b7eaae3ccca9baa3d3e: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 03:20:06.131158 3042 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 96a8981e1d594b7eaae3ccca9baa3d3e: Bootstrap complete.
I20260929 03:20:06.133561 3042 raft_consensus.cc:348] T 00000000000000000000000000000000 P 96a8981e1d594b7eaae3ccca9baa3d3e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96a8981e1d594b7eaae3ccca9baa3d3e" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 33493 } }
I20260929 03:20:06.133813 3042 raft_consensus.cc:729] T 00000000000000000000000000000000 P 96a8981e1d594b7eaae3ccca9baa3d3e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96a8981e1d594b7eaae3ccca9baa3d3e, State: Initialized, Role: FOLLOWER
I20260929 03:20:06.134012 3042 consensus_queue.cc:260] T 00000000000000000000000000000000 P 96a8981e1d594b7eaae3ccca9baa3d3e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96a8981e1d594b7eaae3ccca9baa3d3e" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 33493 } }
I20260929 03:20:06.134095 3042 raft_consensus.cc:388] T 00000000000000000000000000000000 P 96a8981e1d594b7eaae3ccca9baa3d3e [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:20:06.134152 3042 raft_consensus.cc:482] T 00000000000000000000000000000000 P 96a8981e1d594b7eaae3ccca9baa3d3e [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:20:06.134200 3042 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 96a8981e1d594b7eaae3ccca9baa3d3e [term 1 FOLLOWER]: Advancing to term 2
I20260929 03:20:06.135203 3042 raft_consensus.cc:504] T 00000000000000000000000000000000 P 96a8981e1d594b7eaae3ccca9baa3d3e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96a8981e1d594b7eaae3ccca9baa3d3e" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 33493 } }
I20260929 03:20:06.135321 3042 leader_election.cc:302] T 00000000000000000000000000000000 P 96a8981e1d594b7eaae3ccca9baa3d3e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 96a8981e1d594b7eaae3ccca9baa3d3e; no voters:
I20260929 03:20:06.135532 3042 leader_election.cc:288] T 00000000000000000000000000000000 P 96a8981e1d594b7eaae3ccca9baa3d3e [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 03:20:06.135560 3046 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 96a8981e1d594b7eaae3ccca9baa3d3e [term 2 FOLLOWER]: Leader election won for term 2
I20260929 03:20:06.135674 3046 raft_consensus.cc:686] T 00000000000000000000000000000000 P 96a8981e1d594b7eaae3ccca9baa3d3e [term 2 LEADER]: Becoming Leader. State: Replica: 96a8981e1d594b7eaae3ccca9baa3d3e, State: Running, Role: LEADER
I20260929 03:20:06.135808 3042 sys_catalog.cc:564] T 00000000000000000000000000000000 P 96a8981e1d594b7eaae3ccca9baa3d3e [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:20:06.135828 3046 consensus_queue.cc:237] T 00000000000000000000000000000000 P 96a8981e1d594b7eaae3ccca9baa3d3e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96a8981e1d594b7eaae3ccca9baa3d3e" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 33493 } }
I20260929 03:20:06.136234 3048 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96a8981e1d594b7eaae3ccca9baa3d3e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 96a8981e1d594b7eaae3ccca9baa3d3e. Latest consensus state: current_term: 2 leader_uuid: "96a8981e1d594b7eaae3ccca9baa3d3e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96a8981e1d594b7eaae3ccca9baa3d3e" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 33493 } } }
I20260929 03:20:06.136322 3048 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96a8981e1d594b7eaae3ccca9baa3d3e [sys.catalog]: This master's current role is: LEADER
I20260929 03:20:06.136535 3047 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96a8981e1d594b7eaae3ccca9baa3d3e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "96a8981e1d594b7eaae3ccca9baa3d3e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96a8981e1d594b7eaae3ccca9baa3d3e" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 33493 } } }
I20260929 03:20:06.136615 3047 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96a8981e1d594b7eaae3ccca9baa3d3e [sys.catalog]: This master's current role is: LEADER
I20260929 03:20:06.137871 3062 catalog_manager.cc:1259] Loaded cluster ID: 7ac3c4c2fd4743c2a77a251dd9b6ecd1
I20260929 03:20:06.137917 3062 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 96a8981e1d594b7eaae3ccca9baa3d3e: loading cluster ID for follower catalog manager: success
W20260929 03:20:06.138116 3062 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 96a8981e1d594b7eaae3ccca9baa3d3e: acquiring CA information for follower catalog manager: Runtime error: could not decrypt private key with the password returned by the configured command: unable to load data from memory: error reading private key: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D07803A:asn1 encoding routines:ASN1_ITEM_EX_D2I:nested asn1 error:tasn_dec.c:386:Type=X509_ALGOR error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=algor, Type=X509_SIG
W20260929 03:20:06.138159 3062 catalog_manager.cc:873] Runtime error: could not decrypt private key with the password returned by the configured command: unable to load data from memory: error reading private key: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D07803A:asn1 encoding routines:ASN1_ITEM_EX_D2I:nested asn1 error:tasn_dec.c:386:Type=X509_ALGOR error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=algor, Type=X509_SIG: failed to prepare follower catalog manager, will retry
I20260929 03:20:06.138199 3053 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:20:06.138532 3053 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:20:06.138629 3053 catalog_manager.cc:1259] Loaded cluster ID: 7ac3c4c2fd4743c2a77a251dd9b6ecd1
I20260929 03:20:06.138661 3053 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:20:06.138819 3053 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260929 03:20:06.142132 3053 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260929 03:20:06.144483 3053 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:20:06.144706 3053 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 96a8981e1d594b7eaae3ccca9baa3d3e: Loaded TSK: 0
I20260929 03:20:06.144942 3053 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 03:20:06.204535 3044 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:20:06.204824 3044 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:20:06.204865 3044 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:20:06.206130 3044 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:20:06.206187 3044 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:20:06.206228 3044 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.178.65
I20260929 03:20:06.207767 3044 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.178.84:42811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.65.keytab
--principal=kudu/127.20.178.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.178.65:44525
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.178.65
--webserver_port=44209
--tserver_master_addrs=127.20.178.126:33493
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3044
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.178.65
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:20:06.208029 3044 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:20:06.208259 3044 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:20:06.214131 3069 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:20:06.214143 3070 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:20:06.214349 3044 server_base.cc:1034] running on GCE node
W20260929 03:20:06.214157 3072 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:20:06.214641 3044 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:20:06.214929 3044 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:20:06.216120 3044 hybrid_clock.cc:648] HybridClock initialized: now 1790652006216106 us; error 30 us; skew 500 ppm
Sep 29 03:20:06 dist-test-slave-c9qh krb5kdc[2425](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652006, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:20:06.219074 3044 init.cc:377] Logged in from keytab as kudu/127.20.178.65@KRBTEST.COM (short username kudu)
I20260929 03:20:06.219578 3044 webserver.cc:460] Webserver started at http://127.20.178.65:44209/ using document root <none> and password file <none>
I20260929 03:20:06.219810 3044 fs_manager.cc:359] Metadata directory not provided
I20260929 03:20:06.219863 3044 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:20:06.221436 3044 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.000s
I20260929 03:20:06.224301 3079 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:20:06.224510 3044 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 03:20:06.224622 3044 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/minicluster-data/ts-0/data,/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/minicluster-data/ts-0/wal
uuid: "5071e4d22012499c9e198b9b39515856"
format_stamp: "Formatted at 2026-09-29 03:20:05 on dist-test-slave-c9qh"
server_key: "3161551a5e4bfdf56ab8fb970e6b2257"
server_key_iv: "bcd44d9111c99f5686d8b9585d62fe55"
server_key_version: "encryptionkey@0"
I20260929 03:20:06.225083 3044 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:20:06.231981 3044 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:20:06.234297 3044 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:20:06.234462 3044 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:20:06.234709 3044 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:20:06.235159 3086 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260929 03:20:06.237286 3044 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260929 03:20:06.237360 3044 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260929 03:20:06.237407 3044 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260929 03:20:06.238037 3044 ts_tablet_manager.cc:610] Registered 1 tablets
I20260929 03:20:06.238092 3044 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260929 03:20:06.238154 3086 tablet_bootstrap.cc:492] T 8b1ff8fa9d954a688969cd4053496009 P 5071e4d22012499c9e198b9b39515856: Bootstrap starting.
I20260929 03:20:06.244695 3044 rpc_server.cc:307] RPC server started. Bound to: 127.20.178.65:44525
I20260929 03:20:06.245208 3044 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/minicluster-data/ts-0/data/info.pb
I20260929 03:20:06.245638 3193 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.178.65:44525 every 8 connection(s)
I20260929 03:20:06.246474 21193 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuoaHbX/build/release/bin/kudu as pid 3044
I20260929 03:20:06.246755 21193 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuoaHbX/build/release/bin/kudu
/tmp/dist-test-taskuoaHbX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.178.66:37709
--local_ip_for_outbound_sockets=127.20.178.66
--tserver_master_addrs=127.20.178.126:33493
--webserver_port=36125
--webserver_interface=127.20.178.66
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.66.keytab
--principal=kudu/127.20.178.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.178.84:42811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 03:20:06.249154 3086 log.cc:824] T 8b1ff8fa9d954a688969cd4053496009 P 5071e4d22012499c9e198b9b39515856: Log is configured to *not* fsync() on all Append() calls
I20260929 03:20:06.251875 3086 tablet_bootstrap.cc:492] T 8b1ff8fa9d954a688969cd4053496009 P 5071e4d22012499c9e198b9b39515856: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 03:20:06.252183 3086 tablet_bootstrap.cc:492] T 8b1ff8fa9d954a688969cd4053496009 P 5071e4d22012499c9e198b9b39515856: Bootstrap complete.
I20260929 03:20:06.253933 3086 ts_tablet_manager.cc:1397] T 8b1ff8fa9d954a688969cd4053496009 P 5071e4d22012499c9e198b9b39515856: Time spent bootstrapping tablet: real 0.016s user 0.010s sys 0.004s
I20260929 03:20:06.255645 3086 raft_consensus.cc:348] T 8b1ff8fa9d954a688969cd4053496009 P 5071e4d22012499c9e198b9b39515856 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fdb3aa2f42046dda60d96844a22c272" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 37489 } } peers { permanent_uuid: "c4d16a71b9b44e9f95766d5ed5ede8b9" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 37709 } } peers { permanent_uuid: "5071e4d22012499c9e198b9b39515856" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 44525 } }
I20260929 03:20:06.255998 3086 raft_consensus.cc:729] T 8b1ff8fa9d954a688969cd4053496009 P 5071e4d22012499c9e198b9b39515856 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5071e4d22012499c9e198b9b39515856, State: Initialized, Role: FOLLOWER
I20260929 03:20:06.256170 3086 consensus_queue.cc:260] T 8b1ff8fa9d954a688969cd4053496009 P 5071e4d22012499c9e198b9b39515856 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fdb3aa2f42046dda60d96844a22c272" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 37489 } } peers { permanent_uuid: "c4d16a71b9b44e9f95766d5ed5ede8b9" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 37709 } } peers { permanent_uuid: "5071e4d22012499c9e198b9b39515856" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 44525 } }
I20260929 03:20:06.256474 3086 ts_tablet_manager.cc:1428] T 8b1ff8fa9d954a688969cd4053496009 P 5071e4d22012499c9e198b9b39515856: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
Sep 29 03:20:06 dist-test-slave-c9qh krb5kdc[2425](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652006, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.65@KRBTEST.COM for kudu/127.20.178.126@KRBTEST.COM
I20260929 03:20:06.268730 3198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:20:06.252860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:33493 (local address 127.20.178.65:35003)
0929 03:20:06.256445 (+ 3585us) negotiation.cc:107] Waiting for socket to connect
0929 03:20:06.256493 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0929 03:20:06.256809 (+ 316us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:20:06.257448 (+ 639us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:20:06.257452 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:20:06.257861 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:20:06.258184 (+ 323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:20:06.258192 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:20:06.259880 (+ 1688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:20:06.259882 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:20:06.261185 (+ 1303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:20:06.261188 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:20:06.262001 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:20:06.262002 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:20:06.262143 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:20:06.262755 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:20:06.262765 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:20:06.264543 (+ 1778us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:20:06.266880 (+ 2337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:20:06.266887 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:20:06.266889 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:20:06.267190 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:20:06.267460 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:20:06.267461 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:20:06.267461 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:20:06.267612 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:20:06.268036 (+ 424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:20:06.268044 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:20:06.268195 (+ 151us) client_negotiation.cc:769] Sending connection context
0929 03:20:06.268303 (+ 108us) client_negotiation.cc:240] Negotiation successful
0929 03:20:06.268468 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3519,"thread_start_us":81,"threads_started":1}
I20260929 03:20:06.268743 3199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:20:06.252963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.65:35003 (local address 127.20.178.126:33493)
0929 03:20:06.256250 (+ 3287us) server_negotiation.cc:206] Beginning negotiation
0929 03:20:06.256261 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 03:20:06.256686 (+ 425us) server_negotiation.cc:407] Connection header received
0929 03:20:06.257181 (+ 495us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:20:06.257188 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:20:06.257245 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:20:06.257391 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:20:06.258329 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:20:06.259769 (+ 1440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:20:06.261280 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:20:06.261901 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:20:06.262020 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:20:06.264702 (+ 2682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:20:06.264707 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:20:06.264707 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:20:06.264719 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:20:06.266783 (+ 2064us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:20:06.267276 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:20:06.267280 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:20:06.267280 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:20:06.267387 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:20:06.267679 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:20:06.267680 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:20:06.267680 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:20:06.267957 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:20:06.268043 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0929 03:20:06.268366 (+ 323us) server_negotiation.cc:299] Negotiation successful
0929 03:20:06.268498 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3218,"thread_start_us":84,"threads_started":1}
I20260929 03:20:06.269924 3194 heartbeater.cc:347] Connected to a master server at 127.20.178.126:33493
I20260929 03:20:06.270032 3194 heartbeater.cc:464] Registering TS with master...
I20260929 03:20:06.270247 3194 heartbeater.cc:511] Master 127.20.178.126:33493 requested a full tablet report, sending...
I20260929 03:20:06.271070 3006 ts_manager.cc:194] Registered new tserver with Master: 5071e4d22012499c9e198b9b39515856 (127.20.178.65:44525)
I20260929 03:20:06.272207 3006 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.178.65@KRBTEST.COM'} at 127.20.178.65:35003
I20260929 03:20:06.273236 3194 heartbeater.cc:503] Master 127.20.178.126:33493 was elected leader, sending a full tablet report...
Sep 29 03:20:06 dist-test-slave-c9qh krb5kdc[2425](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652006, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.126@KRBTEST.COM for kudu/127.20.178.65@KRBTEST.COM
I20260929 03:20:06.282505 3201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:20:06.271671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.65:44525 (local address 127.0.0.1:55778)
0929 03:20:06.272156 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0929 03:20:06.272191 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0929 03:20:06.272253 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:20:06.273509 (+ 1256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:20:06.273516 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:20:06.273858 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:20:06.273918 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:20:06.273921 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:20:06.275177 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:20:06.275178 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:20:06.276072 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:20:06.276074 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:20:06.276799 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:20:06.276800 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:20:06.276873 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:20:06.277388 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:20:06.277393 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:20:06.279047 (+ 1654us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:20:06.281316 (+ 2269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:20:06.281322 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:20:06.281324 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:20:06.281580 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:20:06.281808 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:20:06.281809 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:20:06.281809 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:20:06.281869 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:20:06.282178 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:20:06.282183 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:20:06.282229 (+ 46us) client_negotiation.cc:769] Sending connection context
0929 03:20:06.282299 (+ 70us) client_negotiation.cc:240] Negotiation successful
0929 03:20:06.282404 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":178,"threads_started":1}
I20260929 03:20:06.283138 3202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:20:06.272007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55778 (local address 127.20.178.65:44525)
0929 03:20:06.272393 (+ 386us) server_negotiation.cc:206] Beginning negotiation
0929 03:20:06.272398 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:20:06.272407 (+ 9us) server_negotiation.cc:407] Connection header received
0929 03:20:06.272525 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:20:06.272527 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:20:06.272541 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:20:06.272594 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:20:06.274012 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:20:06.275085 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:20:06.276161 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:20:06.276682 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:20:06.276744 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:20:06.279141 (+ 2397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:20:06.279148 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:20:06.279148 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:20:06.279155 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:20:06.281207 (+ 2052us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:20:06.281674 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:20:06.281677 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:20:06.281677 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:20:06.281739 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:20:06.281930 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:20:06.281930 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:20:06.281931 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:20:06.282083 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:20:06.282151 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0929 03:20:06.282941 (+ 790us) server_negotiation.cc:299] Negotiation successful
0929 03:20:06.283038 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":67,"threads_started":1}
I20260929 03:20:06.283771 3128 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b1ff8fa9d954a688969cd4053496009 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:20:05 UTC) from {username='kudu', principal='kudu/127.20.178.126@KRBTEST.COM'} at 127.0.0.1:55778
I20260929 03:20:06.284428 3203 tablet_replica.cc:330] stopping tablet replica
I20260929 03:20:06.284533 3203 raft_consensus.cc:2229] T 8b1ff8fa9d954a688969cd4053496009 P 5071e4d22012499c9e198b9b39515856 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:20:06.284600 3203 raft_consensus.cc:2258] T 8b1ff8fa9d954a688969cd4053496009 P 5071e4d22012499c9e198b9b39515856 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:20:06.284957 3203 ts_tablet_manager.cc:1905] T 8b1ff8fa9d954a688969cd4053496009 P 5071e4d22012499c9e198b9b39515856: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:20:06.287372 3203 ts_tablet_manager.cc:1918] T 8b1ff8fa9d954a688969cd4053496009 P 5071e4d22012499c9e198b9b39515856: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:20:06.287446 3203 log.cc:1196] T 8b1ff8fa9d954a688969cd4053496009 P 5071e4d22012499c9e198b9b39515856: Deleting WAL directory at /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/minicluster-data/ts-0/wal/wals/8b1ff8fa9d954a688969cd4053496009
I20260929 03:20:06.287670 3203 ts_tablet_manager.cc:1939] T 8b1ff8fa9d954a688969cd4053496009 P 5071e4d22012499c9e198b9b39515856: Deleting consensus metadata
I20260929 03:20:06.288185 2992 catalog_manager.cc:4841] TS 5071e4d22012499c9e198b9b39515856 (127.20.178.65:44525): tablet 8b1ff8fa9d954a688969cd4053496009 (table test-table [id=39e856c9a3c24348bedae2bd3330212e]) successfully deleted
W20260929 03:20:06.340170 3196 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:20:06.340389 3196 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:20:06.340415 3196 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:20:06.341617 3196 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:20:06.341660 3196 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:20:06.341686 3196 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.178.66
I20260929 03:20:06.343070 3196 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.178.84:42811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.66.keytab
--principal=kudu/127.20.178.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.178.66:37709
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.178.66
--webserver_port=36125
--tserver_master_addrs=127.20.178.126:33493
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3196
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.178.66
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:20:06.343262 3196 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:20:06.343454 3196 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:20:06.348590 3209 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:20:06.348575 3211 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:20:06.348575 3208 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:20:06.348604 3196 server_base.cc:1034] running on GCE node
I20260929 03:20:06.349040 3196 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:20:06.349285 3196 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:20:06.350427 3196 hybrid_clock.cc:648] HybridClock initialized: now 1790652006350416 us; error 33 us; skew 500 ppm
Sep 29 03:20:06 dist-test-slave-c9qh krb5kdc[2425](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652006, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:20:06.352917 3196 init.cc:377] Logged in from keytab as kudu/127.20.178.66@KRBTEST.COM (short username kudu)
I20260929 03:20:06.353360 3196 webserver.cc:460] Webserver started at http://127.20.178.66:36125/ using document root <none> and password file <none>
I20260929 03:20:06.353592 3196 fs_manager.cc:359] Metadata directory not provided
I20260929 03:20:06.353677 3196 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:20:06.354992 3196 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 03:20:06.355571 3218 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:20:06.355747 3196 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260929 03:20:06.355841 3196 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/minicluster-data/ts-1/data,/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/minicluster-data/ts-1/wal
uuid: "c4d16a71b9b44e9f95766d5ed5ede8b9"
format_stamp: "Formatted at 2026-09-29 03:20:05 on dist-test-slave-c9qh"
server_key: "53dbba98e9bfa97db74cadca73323601"
server_key_iv: "696b9603205ac7a46e23303ea2764a6f"
server_key_version: "encryptionkey@0"
I20260929 03:20:06.356204 3196 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:20:06.378219 3196 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:20:06.380188 3196 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:20:06.380362 3196 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:20:06.380625 3196 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:20:06.380985 3196 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:20:06.381040 3196 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:20:06.381083 3196 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:20:06.381114 3196 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:20:06.386870 3196 rpc_server.cc:307] RPC server started. Bound to: 127.20.178.66:37709
I20260929 03:20:06.386948 3331 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.178.66:37709 every 8 connection(s)
I20260929 03:20:06.387300 3196 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/minicluster-data/ts-1/data/info.pb
Sep 29 03:20:06 dist-test-slave-c9qh krb5kdc[2425](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652006, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.66@KRBTEST.COM for kudu/127.20.178.126@KRBTEST.COM
I20260929 03:20:06.395576 21193 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuoaHbX/build/release/bin/kudu as pid 3196
I20260929 03:20:06.395840 21193 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuoaHbX/build/release/bin/kudu
/tmp/dist-test-taskuoaHbX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.178.67:37489
--local_ip_for_outbound_sockets=127.20.178.67
--tserver_master_addrs=127.20.178.126:33493
--webserver_port=34703
--webserver_interface=127.20.178.67
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.67.keytab
--principal=kudu/127.20.178.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.178.84:42811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 03:20:06.396387 3199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:20:06.388045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.66:44789 (local address 127.20.178.126:33493)
0929 03:20:06.388088 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0929 03:20:06.388090 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:20:06.388315 (+ 225us) server_negotiation.cc:407] Connection header received
0929 03:20:06.388876 (+ 561us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:20:06.388877 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:20:06.388894 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:20:06.388955 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:20:06.389884 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:20:06.390401 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:20:06.391348 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:20:06.391562 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:20:06.391619 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:20:06.393761 (+ 2142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:20:06.393762 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:20:06.393762 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:20:06.393768 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:20:06.394952 (+ 1184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:20:06.395302 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:20:06.395303 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:20:06.395303 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:20:06.395354 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:20:06.395604 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:20:06.395606 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:20:06.395606 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:20:06.395775 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:20:06.395843 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0929 03:20:06.396200 (+ 357us) server_negotiation.cc:299] Negotiation successful
0929 03:20:06.396287 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260929 03:20:06.396590 3334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:20:06.388031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:33493 (local address 127.20.178.66:44789)
0929 03:20:06.388197 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0929 03:20:06.388240 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 03:20:06.388793 (+ 553us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:20:06.389032 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:20:06.389035 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:20:06.389597 (+ 562us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:20:06.389791 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:20:06.389795 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:20:06.390480 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:20:06.390481 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:20:06.391267 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:20:06.391269 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:20:06.391640 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:20:06.391640 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:20:06.391735 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:20:06.392280 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:20:06.392289 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:20:06.393684 (+ 1395us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:20:06.395025 (+ 1341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:20:06.395031 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:20:06.395032 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:20:06.395237 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:20:06.395431 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:20:06.395431 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:20:06.395431 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:20:06.395527 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:20:06.395848 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:20:06.395855 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:20:06.396035 (+ 180us) client_negotiation.cc:769] Sending connection context
0929 03:20:06.396167 (+ 132us) client_negotiation.cc:240] Negotiation successful
0929 03:20:06.396326 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":89,"threads_started":1}
I20260929 03:20:06.397109 3332 heartbeater.cc:347] Connected to a master server at 127.20.178.126:33493
I20260929 03:20:06.397223 3332 heartbeater.cc:464] Registering TS with master...
I20260929 03:20:06.397400 3332 heartbeater.cc:511] Master 127.20.178.126:33493 requested a full tablet report, sending...
I20260929 03:20:06.397780 3006 ts_manager.cc:194] Registered new tserver with Master: c4d16a71b9b44e9f95766d5ed5ede8b9 (127.20.178.66:37709)
I20260929 03:20:06.398161 3006 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.178.66@KRBTEST.COM'} at 127.20.178.66:44789
W20260929 03:20:06.477924 3335 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:20:06.478122 3335 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:20:06.478145 3335 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:20:06.479231 3335 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:20:06.479265 3335 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:20:06.479291 3335 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.178.67
I20260929 03:20:06.480783 3335 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.178.84:42811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.67.keytab
--principal=kudu/127.20.178.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.178.67:37489
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.178.67
--webserver_port=34703
--tserver_master_addrs=127.20.178.126:33493
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3335
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.178.67
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:20:06.481015 3335 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:20:06.481189 3335 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:20:06.486140 3340 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:20:06.486146 3343 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:20:06.486158 3341 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:20:06.486470 3335 server_base.cc:1034] running on GCE node
I20260929 03:20:06.486678 3335 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:20:06.486915 3335 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:20:06.488071 3335 hybrid_clock.cc:648] HybridClock initialized: now 1790652006488054 us; error 29 us; skew 500 ppm
Sep 29 03:20:06 dist-test-slave-c9qh krb5kdc[2425](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652006, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:20:06.490564 3335 init.cc:377] Logged in from keytab as kudu/127.20.178.67@KRBTEST.COM (short username kudu)
I20260929 03:20:06.491037 3335 webserver.cc:460] Webserver started at http://127.20.178.67:34703/ using document root <none> and password file <none>
I20260929 03:20:06.491257 3335 fs_manager.cc:359] Metadata directory not provided
I20260929 03:20:06.491334 3335 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:20:06.492725 3335 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 03:20:06.493305 3350 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:20:06.493468 3335 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 03:20:06.493566 3335 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/minicluster-data/ts-2/data,/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/minicluster-data/ts-2/wal
uuid: "0fdb3aa2f42046dda60d96844a22c272"
format_stamp: "Formatted at 2026-09-29 03:20:05 on dist-test-slave-c9qh"
server_key: "5e809bd46f719d2e6cf2f84c6b87e4d6"
server_key_iv: "2d588bc3d30278eb60f5ddc3fa8b86c0"
server_key_version: "encryptionkey@0"
I20260929 03:20:06.493912 3335 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:20:06.504417 3335 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:20:06.506798 3335 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:20:06.506966 3335 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:20:06.507198 3335 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:20:06.507501 3335 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:20:06.507553 3335 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:20:06.507589 3335 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:20:06.507616 3335 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:20:06.513408 3335 rpc_server.cc:307] RPC server started. Bound to: 127.20.178.67:37489
I20260929 03:20:06.513445 3463 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.178.67:37489 every 8 connection(s)
I20260929 03:20:06.513800 3335 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/minicluster-data/ts-2/data/info.pb
Sep 29 03:20:06 dist-test-slave-c9qh krb5kdc[2425](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652006, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.67@KRBTEST.COM for kudu/127.20.178.126@KRBTEST.COM
I20260929 03:20:06.521391 21193 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuoaHbX/build/release/bin/kudu as pid 3335
I20260929 03:20:06.523545 3199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:20:06.514538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.67:57077 (local address 127.20.178.126:33493)
0929 03:20:06.514583 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0929 03:20:06.514585 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:20:06.514810 (+ 225us) server_negotiation.cc:407] Connection header received
0929 03:20:06.515273 (+ 463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:20:06.515274 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:20:06.515284 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:20:06.515335 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:20:06.516515 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:20:06.517106 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:20:06.518063 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:20:06.518252 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:20:06.518308 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:20:06.520506 (+ 2198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:20:06.520507 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:20:06.520508 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:20:06.520515 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:20:06.522058 (+ 1543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:20:06.522541 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:20:06.522542 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:20:06.522543 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:20:06.522602 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:20:06.522893 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:20:06.522894 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:20:06.522895 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:20:06.523033 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:20:06.523104 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0929 03:20:06.523357 (+ 253us) server_negotiation.cc:299] Negotiation successful
0929 03:20:06.523452 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260929 03:20:06.523620 3466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:20:06.514543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:33493 (local address 127.20.178.67:57077)
0929 03:20:06.514687 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0929 03:20:06.514721 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0929 03:20:06.515190 (+ 469us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:20:06.515389 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:20:06.515393 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:20:06.515954 (+ 561us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:20:06.516390 (+ 436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:20:06.516396 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:20:06.517190 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:20:06.517190 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:20:06.517966 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:20:06.517968 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:20:06.518333 (+ 365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:20:06.518334 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:20:06.518436 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:20:06.519034 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:20:06.519043 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:20:06.520391 (+ 1348us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:20:06.522127 (+ 1736us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:20:06.522134 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:20:06.522136 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:20:06.522445 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:20:06.522677 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:20:06.522677 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:20:06.522678 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:20:06.522826 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:20:06.523101 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:20:06.523106 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:20:06.523224 (+ 118us) client_negotiation.cc:769] Sending connection context
0929 03:20:06.523310 (+ 86us) client_negotiation.cc:240] Negotiation successful
0929 03:20:06.523427 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":83,"threads_started":1}
I20260929 03:20:06.524196 3464 heartbeater.cc:347] Connected to a master server at 127.20.178.126:33493
I20260929 03:20:06.524307 3464 heartbeater.cc:464] Registering TS with master...
I20260929 03:20:06.524485 3464 heartbeater.cc:511] Master 127.20.178.126:33493 requested a full tablet report, sending...
I20260929 03:20:06.524928 3006 ts_manager.cc:194] Registered new tserver with Master: 0fdb3aa2f42046dda60d96844a22c272 (127.20.178.67:37489)
I20260929 03:20:06.525399 3006 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.178.67@KRBTEST.COM'} at 127.20.178.67:57077
I20260929 03:20:06.528896 3468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:20:06.521787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56760 (local address 127.20.178.126:33493)
0929 03:20:06.521925 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0929 03:20:06.521926 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 03:20:06.521940 (+ 14us) server_negotiation.cc:407] Connection header received
0929 03:20:06.521995 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:20:06.521996 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:20:06.522009 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:20:06.522086 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:20:06.522837 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:20:06.523624 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:20:06.524966 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:20:06.525257 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:20:06.525319 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:20:06.526534 (+ 1215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:20:06.526535 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:20:06.526536 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:20:06.526547 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:20:06.527760 (+ 1213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:20:06.528154 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:20:06.528154 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:20:06.528155 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:20:06.528205 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:20:06.528435 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:20:06.528435 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:20:06.528436 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:20:06.528530 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:20:06.528689 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0929 03:20:06.528767 (+ 78us) server_negotiation.cc:299] Negotiation successful
0929 03:20:06.528831 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":55,"threads_started":1}
I20260929 03:20:06.529646 21193 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 03:20:06.532472 3468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:20:06.529992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56776 (local address 127.20.178.126:33493)
0929 03:20:06.530060 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0929 03:20:06.530062 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:20:06.530152 (+ 90us) server_negotiation.cc:407] Connection header received
0929 03:20:06.530230 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:20:06.530231 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:20:06.530271 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:20:06.530343 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:20:06.530459 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:20:06.530909 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:20:06.531684 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:20:06.531891 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:20:06.531978 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:20:06.532153 (+ 175us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:20:06.532251 (+ 98us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:20:06.532347 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0929 03:20:06.532370 (+ 23us) server_negotiation.cc:299] Negotiation successful
0929 03:20:06.532393 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 03:20:06.532784 3006 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:56776:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 03:20:06.534288 3006 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 03:20:06.536514 3128 tablet_service.cc:1467] Processing CreateTablet for tablet 13a38243527941ff9914cdf90b184502 (DEFAULT_TABLE table=test-table [id=cbbf8028d179448fa76ad202b630f247]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:20:06.536808 3128 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13a38243527941ff9914cdf90b184502. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:20:06.538527 3086 tablet_bootstrap.cc:492] T 13a38243527941ff9914cdf90b184502 P 5071e4d22012499c9e198b9b39515856: Bootstrap starting.
I20260929 03:20:06.538919 3086 tablet_bootstrap.cc:654] T 13a38243527941ff9914cdf90b184502 P 5071e4d22012499c9e198b9b39515856: Neither blocks nor log segments found. Creating new log.
I20260929 03:20:06.539887 3086 tablet_bootstrap.cc:492] T 13a38243527941ff9914cdf90b184502 P 5071e4d22012499c9e198b9b39515856: No bootstrap required, opened a new log
I20260929 03:20:06.539934 3086 ts_tablet_manager.cc:1397] T 13a38243527941ff9914cdf90b184502 P 5071e4d22012499c9e198b9b39515856: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260929 03:20:06.540077 3086 raft_consensus.cc:348] T 13a38243527941ff9914cdf90b184502 P 5071e4d22012499c9e198b9b39515856 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4d16a71b9b44e9f95766d5ed5ede8b9" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 37709 } } peers { permanent_uuid: "0fdb3aa2f42046dda60d96844a22c272" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 37489 } } peers { permanent_uuid: "5071e4d22012499c9e198b9b39515856" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 44525 } }
I20260929 03:20:06.540174 3086 raft_consensus.cc:374] T 13a38243527941ff9914cdf90b184502 P 5071e4d22012499c9e198b9b39515856 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:20:06.540234 3086 raft_consensus.cc:729] T 13a38243527941ff9914cdf90b184502 P 5071e4d22012499c9e198b9b39515856 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5071e4d22012499c9e198b9b39515856, State: Initialized, Role: FOLLOWER
I20260929 03:20:06.540289 3086 consensus_queue.cc:260] T 13a38243527941ff9914cdf90b184502 P 5071e4d22012499c9e198b9b39515856 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4d16a71b9b44e9f95766d5ed5ede8b9" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 37709 } } peers { permanent_uuid: "0fdb3aa2f42046dda60d96844a22c272" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 37489 } } peers { permanent_uuid: "5071e4d22012499c9e198b9b39515856" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 44525 } }
I20260929 03:20:06.540400 3086 ts_tablet_manager.cc:1428] T 13a38243527941ff9914cdf90b184502 P 5071e4d22012499c9e198b9b39515856: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260929 03:20:06.540482 3201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:20:06.536239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.66:37709 (local address 127.0.0.1:44574)
0929 03:20:06.536279 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0929 03:20:06.536290 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 03:20:06.536323 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:20:06.536697 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:20:06.536698 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:20:06.536700 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:20:06.536749 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:20:06.536750 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:20:06.538097 (+ 1347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:20:06.538098 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:20:06.539230 (+ 1132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:20:06.539235 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:20:06.540212 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:20:06.540213 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:20:06.540290 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:20:06.540295 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 03:20:06.540327 (+ 32us) client_negotiation.cc:240] Negotiation successful
0929 03:20:06.540374 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20260929 03:20:06.540535 3472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:20:06.536492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45846 (local address 127.20.178.67:37489)
0929 03:20:06.536850 (+ 358us) server_negotiation.cc:206] Beginning negotiation
0929 03:20:06.536854 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:20:06.536911 (+ 57us) server_negotiation.cc:407] Connection header received
0929 03:20:06.536951 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:20:06.536953 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:20:06.536967 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:20:06.537067 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:20:06.537291 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:20:06.538052 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:20:06.539174 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:20:06.539830 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:20:06.539931 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:20:06.539997 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0929 03:20:06.540413 (+ 416us) server_negotiation.cc:299] Negotiation successful
0929 03:20:06.540455 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":103,"threads_started":1}
I20260929 03:20:06.540691 3470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:20:06.536255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.67:37489 (local address 127.0.0.1:45846)
0929 03:20:06.536712 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0929 03:20:06.536722 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 03:20:06.536778 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:20:06.537166 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:20:06.537167 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:20:06.537169 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:20:06.537209 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:20:06.537211 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:20:06.538150 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:20:06.538151 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:20:06.539086 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:20:06.539089 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:20:06.540023 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:20:06.540024 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:20:06.540088 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:20:06.540102 (+ 14us) client_negotiation.cc:769] Sending connection context
0929 03:20:06.540219 (+ 117us) client_negotiation.cc:240] Negotiation successful
0929 03:20:06.540319 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":86,"threads_started":1}
I20260929 03:20:06.541100 3471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:20:06.536313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44574 (local address 127.20.178.66:37709)
0929 03:20:06.536485 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0929 03:20:06.536491 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:20:06.536501 (+ 10us) server_negotiation.cc:407] Connection header received
0929 03:20:06.536542 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:20:06.536544 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:20:06.536561 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:20:06.536826 (+ 265us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:20:06.536901 (+ 75us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:20:06.537962 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:20:06.539361 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:20:06.540108 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:20:06.540869 (+ 761us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:20:06.540938 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0929 03:20:06.540961 (+ 23us) server_negotiation.cc:299] Negotiation successful
0929 03:20:06.541017 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":88,"threads_started":1}
I20260929 03:20:06.541484 3398 tablet_service.cc:1467] Processing CreateTablet for tablet 13a38243527941ff9914cdf90b184502 (DEFAULT_TABLE table=test-table [id=cbbf8028d179448fa76ad202b630f247]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:20:06.541813 3398 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13a38243527941ff9914cdf90b184502. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:20:06.541742 3266 tablet_service.cc:1467] Processing CreateTablet for tablet 13a38243527941ff9914cdf90b184502 (DEFAULT_TABLE table=test-table [id=cbbf8028d179448fa76ad202b630f247]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:20:06.542039 3266 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13a38243527941ff9914cdf90b184502. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:20:06.544402 3474 tablet_bootstrap.cc:492] T 13a38243527941ff9914cdf90b184502 P 0fdb3aa2f42046dda60d96844a22c272: Bootstrap starting.
I20260929 03:20:06.544750 3475 tablet_bootstrap.cc:492] T 13a38243527941ff9914cdf90b184502 P c4d16a71b9b44e9f95766d5ed5ede8b9: Bootstrap starting.
I20260929 03:20:06.545130 3474 tablet_bootstrap.cc:654] T 13a38243527941ff9914cdf90b184502 P 0fdb3aa2f42046dda60d96844a22c272: Neither blocks nor log segments found. Creating new log.
I20260929 03:20:06.545426 3475 tablet_bootstrap.cc:654] T 13a38243527941ff9914cdf90b184502 P c4d16a71b9b44e9f95766d5ed5ede8b9: Neither blocks nor log segments found. Creating new log.
I20260929 03:20:06.545753 3474 log.cc:824] T 13a38243527941ff9914cdf90b184502 P 0fdb3aa2f42046dda60d96844a22c272: Log is configured to *not* fsync() on all Append() calls
I20260929 03:20:06.546021 3475 log.cc:824] T 13a38243527941ff9914cdf90b184502 P c4d16a71b9b44e9f95766d5ed5ede8b9: Log is configured to *not* fsync() on all Append() calls
I20260929 03:20:06.546583 3474 tablet_bootstrap.cc:492] T 13a38243527941ff9914cdf90b184502 P 0fdb3aa2f42046dda60d96844a22c272: No bootstrap required, opened a new log
I20260929 03:20:06.546638 3474 ts_tablet_manager.cc:1397] T 13a38243527941ff9914cdf90b184502 P 0fdb3aa2f42046dda60d96844a22c272: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260929 03:20:06.546761 3475 tablet_bootstrap.cc:492] T 13a38243527941ff9914cdf90b184502 P c4d16a71b9b44e9f95766d5ed5ede8b9: No bootstrap required, opened a new log
I20260929 03:20:06.546835 3475 ts_tablet_manager.cc:1397] T 13a38243527941ff9914cdf90b184502 P c4d16a71b9b44e9f95766d5ed5ede8b9: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260929 03:20:06.548167 3474 raft_consensus.cc:348] T 13a38243527941ff9914cdf90b184502 P 0fdb3aa2f42046dda60d96844a22c272 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4d16a71b9b44e9f95766d5ed5ede8b9" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 37709 } } peers { permanent_uuid: "0fdb3aa2f42046dda60d96844a22c272" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 37489 } } peers { permanent_uuid: "5071e4d22012499c9e198b9b39515856" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 44525 } }
I20260929 03:20:06.548292 3474 raft_consensus.cc:374] T 13a38243527941ff9914cdf90b184502 P 0fdb3aa2f42046dda60d96844a22c272 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:20:06.548314 3474 raft_consensus.cc:729] T 13a38243527941ff9914cdf90b184502 P 0fdb3aa2f42046dda60d96844a22c272 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fdb3aa2f42046dda60d96844a22c272, State: Initialized, Role: FOLLOWER
I20260929 03:20:06.548442 3474 consensus_queue.cc:260] T 13a38243527941ff9914cdf90b184502 P 0fdb3aa2f42046dda60d96844a22c272 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4d16a71b9b44e9f95766d5ed5ede8b9" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 37709 } } peers { permanent_uuid: "0fdb3aa2f42046dda60d96844a22c272" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 37489 } } peers { permanent_uuid: "5071e4d22012499c9e198b9b39515856" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 44525 } }
I20260929 03:20:06.548353 3475 raft_consensus.cc:348] T 13a38243527941ff9914cdf90b184502 P c4d16a71b9b44e9f95766d5ed5ede8b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4d16a71b9b44e9f95766d5ed5ede8b9" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 37709 } } peers { permanent_uuid: "0fdb3aa2f42046dda60d96844a22c272" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 37489 } } peers { permanent_uuid: "5071e4d22012499c9e198b9b39515856" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 44525 } }
I20260929 03:20:06.548508 3475 raft_consensus.cc:374] T 13a38243527941ff9914cdf90b184502 P c4d16a71b9b44e9f95766d5ed5ede8b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:20:06.548555 3475 raft_consensus.cc:729] T 13a38243527941ff9914cdf90b184502 P c4d16a71b9b44e9f95766d5ed5ede8b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4d16a71b9b44e9f95766d5ed5ede8b9, State: Initialized, Role: FOLLOWER
I20260929 03:20:06.548697 3475 consensus_queue.cc:260] T 13a38243527941ff9914cdf90b184502 P c4d16a71b9b44e9f95766d5ed5ede8b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4d16a71b9b44e9f95766d5ed5ede8b9" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 37709 } } peers { permanent_uuid: "0fdb3aa2f42046dda60d96844a22c272" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 37489 } } peers { permanent_uuid: "5071e4d22012499c9e198b9b39515856" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 44525 } }
I20260929 03:20:06.548755 3474 ts_tablet_manager.cc:1428] T 13a38243527941ff9914cdf90b184502 P 0fdb3aa2f42046dda60d96844a22c272: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260929 03:20:06.548764 3464 heartbeater.cc:503] Master 127.20.178.126:33493 was elected leader, sending a full tablet report...
I20260929 03:20:06.548974 3475 ts_tablet_manager.cc:1428] T 13a38243527941ff9914cdf90b184502 P c4d16a71b9b44e9f95766d5ed5ede8b9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 03:20:06.549033 3332 heartbeater.cc:503] Master 127.20.178.126:33493 was elected leader, sending a full tablet report...
I20260929 03:20:06.571635 3200 raft_consensus.cc:482] T 13a38243527941ff9914cdf90b184502 P 5071e4d22012499c9e198b9b39515856 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:20:06.571810 3200 raft_consensus.cc:504] T 13a38243527941ff9914cdf90b184502 P 5071e4d22012499c9e198b9b39515856 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4d16a71b9b44e9f95766d5ed5ede8b9" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 37709 } } peers { permanent_uuid: "0fdb3aa2f42046dda60d96844a22c272" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 37489 } } peers { permanent_uuid: "5071e4d22012499c9e198b9b39515856" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 44525 } }
I20260929 03:20:06.572148 3200 leader_election.cc:288] T 13a38243527941ff9914cdf90b184502 P 5071e4d22012499c9e198b9b39515856 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c4d16a71b9b44e9f95766d5ed5ede8b9 (127.20.178.66:37709), 0fdb3aa2f42046dda60d96844a22c272 (127.20.178.67:37489)
I20260929 03:20:06.575305 3471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:20:06.572408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.65:58963 (local address 127.20.178.66:37709)
0929 03:20:06.572456 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0929 03:20:06.572458 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:20:06.572469 (+ 11us) server_negotiation.cc:407] Connection header received
0929 03:20:06.572535 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:20:06.572536 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:20:06.572546 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:20:06.572617 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:20:06.572813 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:20:06.573354 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:20:06.574287 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:20:06.574893 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:20:06.574948 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:20:06.574955 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0929 03:20:06.575189 (+ 234us) server_negotiation.cc:299] Negotiation successful
0929 03:20:06.575226 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260929 03:20:06.575306 3472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:20:06.572419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.65:59583 (local address 127.20.178.67:37489)
0929 03:20:06.572460 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0929 03:20:06.572463 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:20:06.572664 (+ 201us) server_negotiation.cc:407] Connection header received
0929 03:20:06.572692 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:20:06.572694 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:20:06.572704 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:20:06.572758 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:20:06.572954 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:20:06.573420 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:20:06.574355 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:20:06.574936 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:20:06.575008 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:20:06.575020 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0929 03:20:06.575188 (+ 168us) server_negotiation.cc:299] Negotiation successful
0929 03:20:06.575228 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260929 03:20:06.575346 3198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:20:06.572332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.66:37709 (local address 127.20.178.65:58963)
0929 03:20:06.572398 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0929 03:20:06.572410 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 03:20:06.572462 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:20:06.572658 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:20:06.572659 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:20:06.572660 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:20:06.572742 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:20:06.572744 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:20:06.573436 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:20:06.573437 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:20:06.574198 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:20:06.574200 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:20:06.574965 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:20:06.574966 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:20:06.575081 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:20:06.575111 (+ 30us) client_negotiation.cc:769] Sending connection context
0929 03:20:06.575181 (+ 70us) client_negotiation.cc:240] Negotiation successful
0929 03:20:06.575231 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42,"mutex_wait_us":3}
I20260929 03:20:06.575563 3480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:20:06.572334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.67:37489 (local address 127.20.178.65:59583)
0929 03:20:06.572548 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0929 03:20:06.572558 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 03:20:06.572615 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:20:06.572790 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:20:06.572791 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:20:06.572793 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:20:06.572878 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:20:06.572880 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:20:06.573501 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:20:06.573502 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:20:06.574264 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:20:06.574267 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:20:06.575039 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:20:06.575040 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:20:06.575107 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:20:06.575111 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 03:20:06.575180 (+ 69us) client_negotiation.cc:240] Negotiation successful
0929 03:20:06.575226 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"mutex_wait_us":23,"thread_start_us":107,"threads_started":1}
I20260929 03:20:06.575711 3286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13a38243527941ff9914cdf90b184502" candidate_uuid: "5071e4d22012499c9e198b9b39515856" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4d16a71b9b44e9f95766d5ed5ede8b9" is_pre_election: true
I20260929 03:20:06.575886 3286 raft_consensus.cc:2454] T 13a38243527941ff9914cdf90b184502 P c4d16a71b9b44e9f95766d5ed5ede8b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5071e4d22012499c9e198b9b39515856 in term 0.
I20260929 03:20:06.575845 3418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13a38243527941ff9914cdf90b184502" candidate_uuid: "5071e4d22012499c9e198b9b39515856" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fdb3aa2f42046dda60d96844a22c272" is_pre_election: true
I20260929 03:20:06.575979 3418 raft_consensus.cc:2454] T 13a38243527941ff9914cdf90b184502 P 0fdb3aa2f42046dda60d96844a22c272 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5071e4d22012499c9e198b9b39515856 in term 0.
I20260929 03:20:06.576121 3080 leader_election.cc:302] T 13a38243527941ff9914cdf90b184502 P 5071e4d22012499c9e198b9b39515856 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5071e4d22012499c9e198b9b39515856, c4d16a71b9b44e9f95766d5ed5ede8b9; no voters:
I20260929 03:20:06.576251 3200 raft_consensus.cc:2781] T 13a38243527941ff9914cdf90b184502 P 5071e4d22012499c9e198b9b39515856 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 03:20:06.576304 3200 raft_consensus.cc:482] T 13a38243527941ff9914cdf90b184502 P 5071e4d22012499c9e198b9b39515856 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 03:20:06.576337 3200 raft_consensus.cc:3037] T 13a38243527941ff9914cdf90b184502 P 5071e4d22012499c9e198b9b39515856 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:20:06.577064 3200 raft_consensus.cc:504] T 13a38243527941ff9914cdf90b184502 P 5071e4d22012499c9e198b9b39515856 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4d16a71b9b44e9f95766d5ed5ede8b9" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 37709 } } peers { permanent_uuid: "0fdb3aa2f42046dda60d96844a22c272" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 37489 } } peers { permanent_uuid: "5071e4d22012499c9e198b9b39515856" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 44525 } }
I20260929 03:20:06.577216 3200 leader_election.cc:288] T 13a38243527941ff9914cdf90b184502 P 5071e4d22012499c9e198b9b39515856 [CANDIDATE]: Term 1 election: Requested vote from peers c4d16a71b9b44e9f95766d5ed5ede8b9 (127.20.178.66:37709), 0fdb3aa2f42046dda60d96844a22c272 (127.20.178.67:37489)
I20260929 03:20:06.577363 3418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13a38243527941ff9914cdf90b184502" candidate_uuid: "5071e4d22012499c9e198b9b39515856" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fdb3aa2f42046dda60d96844a22c272"
I20260929 03:20:06.577374 3286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13a38243527941ff9914cdf90b184502" candidate_uuid: "5071e4d22012499c9e198b9b39515856" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4d16a71b9b44e9f95766d5ed5ede8b9"
I20260929 03:20:06.577440 3418 raft_consensus.cc:3037] T 13a38243527941ff9914cdf90b184502 P 0fdb3aa2f42046dda60d96844a22c272 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:20:06.577461 3286 raft_consensus.cc:3037] T 13a38243527941ff9914cdf90b184502 P c4d16a71b9b44e9f95766d5ed5ede8b9 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:20:06.578176 3418 raft_consensus.cc:2454] T 13a38243527941ff9914cdf90b184502 P 0fdb3aa2f42046dda60d96844a22c272 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5071e4d22012499c9e198b9b39515856 in term 1.
I20260929 03:20:06.578183 3286 raft_consensus.cc:2454] T 13a38243527941ff9914cdf90b184502 P c4d16a71b9b44e9f95766d5ed5ede8b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5071e4d22012499c9e198b9b39515856 in term 1.
I20260929 03:20:06.578353 3081 leader_election.cc:302] T 13a38243527941ff9914cdf90b184502 P 5071e4d22012499c9e198b9b39515856 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fdb3aa2f42046dda60d96844a22c272, 5071e4d22012499c9e198b9b39515856; no voters:
I20260929 03:20:06.578451 3200 raft_consensus.cc:2781] T 13a38243527941ff9914cdf90b184502 P 5071e4d22012499c9e198b9b39515856 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:20:06.578605 3200 raft_consensus.cc:686] T 13a38243527941ff9914cdf90b184502 P 5071e4d22012499c9e198b9b39515856 [term 1 LEADER]: Becoming Leader. State: Replica: 5071e4d22012499c9e198b9b39515856, State: Running, Role: LEADER
I20260929 03:20:06.578689 3200 consensus_queue.cc:237] T 13a38243527941ff9914cdf90b184502 P 5071e4d22012499c9e198b9b39515856 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4d16a71b9b44e9f95766d5ed5ede8b9" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 37709 } } peers { permanent_uuid: "0fdb3aa2f42046dda60d96844a22c272" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 37489 } } peers { permanent_uuid: "5071e4d22012499c9e198b9b39515856" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 44525 } }
I20260929 03:20:06.579469 3006 catalog_manager.cc:5495] T 13a38243527941ff9914cdf90b184502 P 5071e4d22012499c9e198b9b39515856 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5071e4d22012499c9e198b9b39515856 (127.20.178.65). New cstate: current_term: 1 leader_uuid: "5071e4d22012499c9e198b9b39515856" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4d16a71b9b44e9f95766d5ed5ede8b9" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 37709 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0fdb3aa2f42046dda60d96844a22c272" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 37489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5071e4d22012499c9e198b9b39515856" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 44525 } health_report { overall_health: HEALTHY } } }
I20260929 03:20:06.588196 3202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:20:06.586015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55786 (local address 127.20.178.65:44525)
0929 03:20:06.586080 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0929 03:20:06.586082 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:20:06.586091 (+ 9us) server_negotiation.cc:407] Connection header received
0929 03:20:06.586159 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:20:06.586159 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:20:06.586174 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:20:06.586225 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:20:06.586344 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:20:06.586799 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:20:06.587458 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:20:06.587651 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:20:06.587700 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:20:06.587856 (+ 156us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:20:06.587960 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:20:06.588047 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0929 03:20:06.588094 (+ 47us) server_negotiation.cc:299] Negotiation successful
0929 03:20:06.588116 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260929 03:20:06.589653 3286 raft_consensus.cc:1261] T 13a38243527941ff9914cdf90b184502 P c4d16a71b9b44e9f95766d5ed5ede8b9 [term 1 FOLLOWER]: Refusing update from remote peer 5071e4d22012499c9e198b9b39515856: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:20:06.590008 3418 raft_consensus.cc:1261] T 13a38243527941ff9914cdf90b184502 P 0fdb3aa2f42046dda60d96844a22c272 [term 1 FOLLOWER]: Refusing update from remote peer 5071e4d22012499c9e198b9b39515856: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:20:06.590014 3200 consensus_queue.cc:1035] T 13a38243527941ff9914cdf90b184502 P 5071e4d22012499c9e198b9b39515856 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4d16a71b9b44e9f95766d5ed5ede8b9" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 37709 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:20:06.590314 3200 consensus_queue.cc:1035] T 13a38243527941ff9914cdf90b184502 P 5071e4d22012499c9e198b9b39515856 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fdb3aa2f42046dda60d96844a22c272" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:20:06.592402 3483 mvcc.cc:205] Tried to move back new op lower bound from 7334510618988871680 to 7334510618946842624. Current Snapshot: MvccSnapshot[applied={T|T < 7334510618988871680}]
I20260929 03:20:06.593375 3484 mvcc.cc:205] Tried to move back new op lower bound from 7334510618988871680 to 7334510618946842624. Current Snapshot: MvccSnapshot[applied={T|T < 7334510618988871680}]
I20260929 03:20:06.594446 3006 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:56776:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:20:06.594523 3006 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:56776:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:20:06.595235 3006 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 96a8981e1d594b7eaae3ccca9baa3d3e: Sending DeleteTablet for 3 replicas of tablet 13a38243527941ff9914cdf90b184502
I20260929 03:20:06.595513 3128 tablet_service.cc:1514] Processing DeleteTablet for tablet 13a38243527941ff9914cdf90b184502 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:20:06 UTC) from {username='kudu', principal='kudu/127.20.178.126@KRBTEST.COM'} at 127.0.0.1:55778
I20260929 03:20:06.595544 3266 tablet_service.cc:1514] Processing DeleteTablet for tablet 13a38243527941ff9914cdf90b184502 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:20:06 UTC) from {username='kudu', principal='kudu/127.20.178.126@KRBTEST.COM'} at 127.0.0.1:44574
I20260929 03:20:06.595606 3203 tablet_replica.cc:330] T 13a38243527941ff9914cdf90b184502 P 5071e4d22012499c9e198b9b39515856: stopping tablet replica
I20260929 03:20:06.595623 21193 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuoaHbX/build/release/bin/kudu with pid 3044
I20260929 03:20:06.595623 3398 tablet_service.cc:1514] Processing DeleteTablet for tablet 13a38243527941ff9914cdf90b184502 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:20:06 UTC) from {username='kudu', principal='kudu/127.20.178.126@KRBTEST.COM'} at 127.0.0.1:45846
I20260929 03:20:06.595657 3203 raft_consensus.cc:2229] T 13a38243527941ff9914cdf90b184502 P 5071e4d22012499c9e198b9b39515856 [term 1 LEADER]: Raft consensus shutting down.
I20260929 03:20:06.595749 3203 raft_consensus.cc:2258] T 13a38243527941ff9914cdf90b184502 P 5071e4d22012499c9e198b9b39515856 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:20:06.596024 3493 tablet_replica.cc:330] T 13a38243527941ff9914cdf90b184502 P c4d16a71b9b44e9f95766d5ed5ede8b9: stopping tablet replica
I20260929 03:20:06.596167 3493 raft_consensus.cc:2229] T 13a38243527941ff9914cdf90b184502 P c4d16a71b9b44e9f95766d5ed5ede8b9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:20:06.596199 3494 tablet_replica.cc:330] T 13a38243527941ff9914cdf90b184502 P 0fdb3aa2f42046dda60d96844a22c272: stopping tablet replica
I20260929 03:20:06.596295 3494 raft_consensus.cc:2229] T 13a38243527941ff9914cdf90b184502 P 0fdb3aa2f42046dda60d96844a22c272 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:20:06.596395 3494 raft_consensus.cc:2258] T 13a38243527941ff9914cdf90b184502 P 0fdb3aa2f42046dda60d96844a22c272 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:20:06.596395 3493 raft_consensus.cc:2258] T 13a38243527941ff9914cdf90b184502 P c4d16a71b9b44e9f95766d5ed5ede8b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:20:06.597030 3494 ts_tablet_manager.cc:1905] T 13a38243527941ff9914cdf90b184502 P 0fdb3aa2f42046dda60d96844a22c272: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:20:06.597134 3493 ts_tablet_manager.cc:1905] T 13a38243527941ff9914cdf90b184502 P c4d16a71b9b44e9f95766d5ed5ede8b9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:20:06.598852 3493 ts_tablet_manager.cc:1918] T 13a38243527941ff9914cdf90b184502 P c4d16a71b9b44e9f95766d5ed5ede8b9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:20:06.598852 3494 ts_tablet_manager.cc:1918] T 13a38243527941ff9914cdf90b184502 P 0fdb3aa2f42046dda60d96844a22c272: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:20:06.598925 3494 log.cc:1196] T 13a38243527941ff9914cdf90b184502 P 0fdb3aa2f42046dda60d96844a22c272: Deleting WAL directory at /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/minicluster-data/ts-2/wal/wals/13a38243527941ff9914cdf90b184502
I20260929 03:20:06.598980 3493 log.cc:1196] T 13a38243527941ff9914cdf90b184502 P c4d16a71b9b44e9f95766d5ed5ede8b9: Deleting WAL directory at /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/minicluster-data/ts-1/wal/wals/13a38243527941ff9914cdf90b184502
I20260929 03:20:06.599257 3494 ts_tablet_manager.cc:1939] T 13a38243527941ff9914cdf90b184502 P 0fdb3aa2f42046dda60d96844a22c272: Deleting consensus metadata
I20260929 03:20:06.599352 3493 ts_tablet_manager.cc:1939] T 13a38243527941ff9914cdf90b184502 P c4d16a71b9b44e9f95766d5ed5ede8b9: Deleting consensus metadata
I20260929 03:20:06.599713 2991 catalog_manager.cc:4841] TS 0fdb3aa2f42046dda60d96844a22c272 (127.20.178.67:37489): tablet 13a38243527941ff9914cdf90b184502 (table test-table [id=cbbf8028d179448fa76ad202b630f247]) successfully deleted
I20260929 03:20:06.599726 2990 catalog_manager.cc:4841] TS c4d16a71b9b44e9f95766d5ed5ede8b9 (127.20.178.66:37709): tablet 13a38243527941ff9914cdf90b184502 (table test-table [id=cbbf8028d179448fa76ad202b630f247]) successfully deleted
W20260929 03:20:06.601579 2992 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.20.178.65:44525 (error 108)
I20260929 03:20:06.601877 21193 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuoaHbX/build/release/bin/kudu with pid 3196
I20260929 03:20:06.602053 3470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:20:06.601935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.65:44525 (local address 127.0.0.1:55796)
0929 03:20:06.601982 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0929 03:20:06.602014 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.178.65:44525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260929 03:20:06.602154 2992 catalog_manager.cc:4584] TS 5071e4d22012499c9e198b9b39515856 (127.20.178.65:44525): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 13a38243527941ff9914cdf90b184502: Network error: Client connection negotiation failed: client connection to 127.20.178.65:44525: connect: Connection refused (error 111)
I20260929 03:20:06.606657 21193 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuoaHbX/build/release/bin/kudu with pid 3335
I20260929 03:20:06.611104 21193 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuoaHbX/build/release/bin/kudu with pid 2973
I20260929 03:20:06.616190 21193 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuoaHbX/build/release/bin/kudu
/tmp/dist-test-taskuoaHbX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.178.126:33493
--webserver_interface=127.20.178.126
--webserver_port=34937
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.126.keytab
--principal=kudu/127.20.178.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.178.84:42811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.178.126:33493
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:20:06.695772 3496 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:20:06.696039 3496 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:20:06.696076 3496 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:20:06.697299 3496 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:20:06.697361 3496 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:20:06.697387 3496 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:20:06.697417 3496 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:20:06.697443 3496 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:20:06.698895 3496 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.178.84:42811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.178.126:33493
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.126.keytab
--principal=kudu/127.20.178.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.178.126:33493
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.178.126
--webserver_port=34937
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3496
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:20:06.699168 3496 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:20:06.699391 3496 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:20:06.704123 3501 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:20:06.704169 3504 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:20:06.704233 3502 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:20:06.704487 3496 server_base.cc:1034] running on GCE node
I20260929 03:20:06.704763 3496 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:20:06.705042 3496 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:20:06.706207 3496 hybrid_clock.cc:648] HybridClock initialized: now 1790652006706200 us; error 35 us; skew 500 ppm
Sep 29 03:20:06 dist-test-slave-c9qh krb5kdc[2425](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652006, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:20:06.708611 3496 init.cc:377] Logged in from keytab as kudu/127.20.178.126@KRBTEST.COM (short username kudu)
I20260929 03:20:06.709124 3496 webserver.cc:460] Webserver started at http://127.20.178.126:34937/ using document root <none> and password file <none>
I20260929 03:20:06.709347 3496 fs_manager.cc:359] Metadata directory not provided
I20260929 03:20:06.709416 3496 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:20:06.710709 3496 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 03:20:06.711344 3511 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:20:06.711513 3496 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 03:20:06.711609 3496 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/minicluster-data/master-0/data,/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/minicluster-data/master-0/wal
uuid: "96a8981e1d594b7eaae3ccca9baa3d3e"
format_stamp: "Formatted at 2026-09-29 03:20:05 on dist-test-slave-c9qh"
server_key: "450b0b8b1ce02ef39fd2c655820f5287"
server_key_iv: "0b98de62dc721901f87586f3ae0b23e2"
server_key_version: "encryptionkey@0"
I20260929 03:20:06.711946 3496 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:20:06.734010 3496 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:20:06.735991 3496 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:20:06.736174 3496 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:20:06.740144 3496 rpc_server.cc:307] RPC server started. Bound to: 127.20.178.126:33493
I20260929 03:20:06.740180 3563 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.178.126:33493 every 8 connection(s)
I20260929 03:20:06.740568 3496 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/minicluster-data/master-0/data/info.pb
I20260929 03:20:06.740831 21193 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuoaHbX/build/release/bin/kudu as pid 3496
I20260929 03:20:06.741110 21193 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuoaHbX/build/release/bin/kudu
/tmp/dist-test-taskuoaHbX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.178.65:44525
--local_ip_for_outbound_sockets=127.20.178.65
--tserver_master_addrs=127.20.178.126:33493
--webserver_port=44209
--webserver_interface=127.20.178.65
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.65.keytab
--principal=kudu/127.20.178.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.178.84:42811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 03:20:06.742131 3564 sys_catalog.cc:263] Verifying existing consensus state
I20260929 03:20:06.742786 3564 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 96a8981e1d594b7eaae3ccca9baa3d3e: Bootstrap starting.
I20260929 03:20:06.750917 3564 log.cc:824] T 00000000000000000000000000000000 P 96a8981e1d594b7eaae3ccca9baa3d3e: Log is configured to *not* fsync() on all Append() calls
I20260929 03:20:06.753176 3564 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 96a8981e1d594b7eaae3ccca9baa3d3e: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 03:20:06.753461 3564 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 96a8981e1d594b7eaae3ccca9baa3d3e: Bootstrap complete.
I20260929 03:20:06.755649 3564 raft_consensus.cc:348] T 00000000000000000000000000000000 P 96a8981e1d594b7eaae3ccca9baa3d3e [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96a8981e1d594b7eaae3ccca9baa3d3e" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 33493 } }
I20260929 03:20:06.755880 3564 raft_consensus.cc:729] T 00000000000000000000000000000000 P 96a8981e1d594b7eaae3ccca9baa3d3e [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96a8981e1d594b7eaae3ccca9baa3d3e, State: Initialized, Role: FOLLOWER
I20260929 03:20:06.756086 3564 consensus_queue.cc:260] T 00000000000000000000000000000000 P 96a8981e1d594b7eaae3ccca9baa3d3e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96a8981e1d594b7eaae3ccca9baa3d3e" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 33493 } }
I20260929 03:20:06.756183 3564 raft_consensus.cc:388] T 00000000000000000000000000000000 P 96a8981e1d594b7eaae3ccca9baa3d3e [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:20:06.756235 3564 raft_consensus.cc:482] T 00000000000000000000000000000000 P 96a8981e1d594b7eaae3ccca9baa3d3e [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:20:06.756285 3564 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 96a8981e1d594b7eaae3ccca9baa3d3e [term 2 FOLLOWER]: Advancing to term 3
I20260929 03:20:06.757294 3564 raft_consensus.cc:504] T 00000000000000000000000000000000 P 96a8981e1d594b7eaae3ccca9baa3d3e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96a8981e1d594b7eaae3ccca9baa3d3e" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 33493 } }
I20260929 03:20:06.757411 3564 leader_election.cc:302] T 00000000000000000000000000000000 P 96a8981e1d594b7eaae3ccca9baa3d3e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 96a8981e1d594b7eaae3ccca9baa3d3e; no voters:
I20260929 03:20:06.757608 3564 leader_election.cc:288] T 00000000000000000000000000000000 P 96a8981e1d594b7eaae3ccca9baa3d3e [CANDIDATE]: Term 3 election: Requested vote from peers
I20260929 03:20:06.757642 3568 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 96a8981e1d594b7eaae3ccca9baa3d3e [term 3 FOLLOWER]: Leader election won for term 3
I20260929 03:20:06.757750 3568 raft_consensus.cc:686] T 00000000000000000000000000000000 P 96a8981e1d594b7eaae3ccca9baa3d3e [term 3 LEADER]: Becoming Leader. State: Replica: 96a8981e1d594b7eaae3ccca9baa3d3e, State: Running, Role: LEADER
I20260929 03:20:06.757836 3568 consensus_queue.cc:237] T 00000000000000000000000000000000 P 96a8981e1d594b7eaae3ccca9baa3d3e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96a8981e1d594b7eaae3ccca9baa3d3e" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 33493 } }
I20260929 03:20:06.757865 3564 sys_catalog.cc:564] T 00000000000000000000000000000000 P 96a8981e1d594b7eaae3ccca9baa3d3e [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:20:06.758257 3570 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96a8981e1d594b7eaae3ccca9baa3d3e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 96a8981e1d594b7eaae3ccca9baa3d3e. Latest consensus state: current_term: 3 leader_uuid: "96a8981e1d594b7eaae3ccca9baa3d3e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96a8981e1d594b7eaae3ccca9baa3d3e" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 33493 } } }
I20260929 03:20:06.758332 3570 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96a8981e1d594b7eaae3ccca9baa3d3e [sys.catalog]: This master's current role is: LEADER
I20260929 03:20:06.758628 3572 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:20:06.759379 3572 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:20:06.759473 3572 catalog_manager.cc:1259] Loaded cluster ID: 7ac3c4c2fd4743c2a77a251dd9b6ecd1
I20260929 03:20:06.759732 3572 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:20:06.759666 3569 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96a8981e1d594b7eaae3ccca9baa3d3e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "96a8981e1d594b7eaae3ccca9baa3d3e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96a8981e1d594b7eaae3ccca9baa3d3e" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 33493 } } }
I20260929 03:20:06.759883 3569 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96a8981e1d594b7eaae3ccca9baa3d3e [sys.catalog]: This master's current role is: LEADER
F20260929 03:20:06.759958 3572 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fa4cfb5a9ad google::LogMessage::Fail() at ??:0
@ 0x7fa4cfb5d88a google::LogMessage::SendToLog() at ??:0
@ 0x7fa4cfb5a4de google::LogMessage::Flush() at ??:0
@ 0x7fa4cfb5ba19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fa4d7a7f0f1 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7fa4d7a888b9 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fa4d1a280d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fa4d1a1d954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fa4d28816db start_thread at ??:0
@ 0x7fa4ce65861f clone at ??:0
W20260929 03:20:06.834159 3566 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:20:06.834347 3566 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:20:06.834370 3566 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:20:06.835489 3566 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:20:06.835536 3566 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:20:06.835563 3566 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.178.65
I20260929 03:20:06.837188 3566 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.178.84:42811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.65.keytab
--principal=kudu/127.20.178.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.178.65:44525
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.178.65
--webserver_port=44209
--tserver_master_addrs=127.20.178.126:33493
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3566
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.178.65
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:20:06.837456 3566 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:20:06.837762 3566 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:20:06.842643 3600 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:20:06.842785 3566 server_base.cc:1034] running on GCE node
W20260929 03:20:06.842643 3603 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:20:06.842864 3601 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:20:06.843050 3566 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:20:06.843317 3566 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:20:06.844482 3566 hybrid_clock.cc:648] HybridClock initialized: now 1790652006844472 us; error 27 us; skew 500 ppm
Sep 29 03:20:06 dist-test-slave-c9qh krb5kdc[2425](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652006, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:20:06.847046 3566 init.cc:377] Logged in from keytab as kudu/127.20.178.65@KRBTEST.COM (short username kudu)
I20260929 03:20:06.847539 3566 webserver.cc:460] Webserver started at http://127.20.178.65:44209/ using document root <none> and password file <none>
I20260929 03:20:06.847751 3566 fs_manager.cc:359] Metadata directory not provided
I20260929 03:20:06.847817 3566 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:20:06.849220 3566 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 03:20:06.849892 3610 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:20:06.850148 3566 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260929 03:20:06.850245 3566 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/minicluster-data/ts-0/data,/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/minicluster-data/ts-0/wal
uuid: "5071e4d22012499c9e198b9b39515856"
format_stamp: "Formatted at 2026-09-29 03:20:05 on dist-test-slave-c9qh"
server_key: "3161551a5e4bfdf56ab8fb970e6b2257"
server_key_iv: "bcd44d9111c99f5686d8b9585d62fe55"
server_key_version: "encryptionkey@0"
I20260929 03:20:06.850658 3566 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:20:06.862387 3566 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:20:06.864707 3566 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:20:06.864866 3566 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:20:06.865103 3566 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:20:06.865540 3617 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260929 03:20:06.867640 3566 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260929 03:20:06.867699 3566 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260929 03:20:06.867746 3566 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260929 03:20:06.868350 3566 ts_tablet_manager.cc:610] Registered 1 tablets
I20260929 03:20:06.868407 3566 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260929 03:20:06.868549 3617 tablet_bootstrap.cc:492] T 13a38243527941ff9914cdf90b184502 P 5071e4d22012499c9e198b9b39515856: Bootstrap starting.
I20260929 03:20:06.875540 3566 rpc_server.cc:307] RPC server started. Bound to: 127.20.178.65:44525
I20260929 03:20:06.875660 3724 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.178.65:44525 every 8 connection(s)
I20260929 03:20:06.875991 3566 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/minicluster-data/ts-0/data/info.pb
I20260929 03:20:06.880584 3617 log.cc:824] T 13a38243527941ff9914cdf90b184502 P 5071e4d22012499c9e198b9b39515856: Log is configured to *not* fsync() on all Append() calls
I20260929 03:20:06.883064 3617 tablet_bootstrap.cc:492] T 13a38243527941ff9914cdf90b184502 P 5071e4d22012499c9e198b9b39515856: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 03:20:06.883342 3617 tablet_bootstrap.cc:492] T 13a38243527941ff9914cdf90b184502 P 5071e4d22012499c9e198b9b39515856: Bootstrap complete.
I20260929 03:20:06.884342 3617 ts_tablet_manager.cc:1397] T 13a38243527941ff9914cdf90b184502 P 5071e4d22012499c9e198b9b39515856: Time spent bootstrapping tablet: real 0.016s user 0.000s sys 0.012s
I20260929 03:20:06.885762 3617 raft_consensus.cc:348] T 13a38243527941ff9914cdf90b184502 P 5071e4d22012499c9e198b9b39515856 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4d16a71b9b44e9f95766d5ed5ede8b9" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 37709 } } peers { permanent_uuid: "0fdb3aa2f42046dda60d96844a22c272" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 37489 } } peers { permanent_uuid: "5071e4d22012499c9e198b9b39515856" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 44525 } }
I20260929 03:20:06.886027 3617 raft_consensus.cc:729] T 13a38243527941ff9914cdf90b184502 P 5071e4d22012499c9e198b9b39515856 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5071e4d22012499c9e198b9b39515856, State: Initialized, Role: FOLLOWER
I20260929 03:20:06.886134 21193 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuoaHbX/build/release/bin/kudu as pid 3566
I20260929 03:20:06.886183 3617 consensus_queue.cc:260] T 13a38243527941ff9914cdf90b184502 P 5071e4d22012499c9e198b9b39515856 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4d16a71b9b44e9f95766d5ed5ede8b9" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 37709 } } peers { permanent_uuid: "0fdb3aa2f42046dda60d96844a22c272" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 37489 } } peers { permanent_uuid: "5071e4d22012499c9e198b9b39515856" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 44525 } }
I20260929 03:20:06.886457 21193 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuoaHbX/build/release/bin/kudu
/tmp/dist-test-taskuoaHbX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.178.66:37709
--local_ip_for_outbound_sockets=127.20.178.66
--tserver_master_addrs=127.20.178.126:33493
--webserver_port=36125
--webserver_interface=127.20.178.66
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.66.keytab
--principal=kudu/127.20.178.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.178.84:42811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 03:20:06.886473 3617 ts_tablet_manager.cc:1428] T 13a38243527941ff9914cdf90b184502 P 5071e4d22012499c9e198b9b39515856: Time spent starting tablet: real 0.002s user 0.002s sys 0.004s
W20260929 03:20:06.967142 3730 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:20:06.967356 3730 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:20:06.967393 3730 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:20:06.968511 3730 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:20:06.968573 3730 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:20:06.968617 3730 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.178.66
I20260929 03:20:06.970047 3730 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.178.84:42811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.66.keytab
--principal=kudu/127.20.178.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.178.66:37709
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.178.66
--webserver_port=36125
--tserver_master_addrs=127.20.178.126:33493
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3730
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.178.66
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:20:06.970296 3730 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:20:06.970516 3730 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:20:06.975275 3735 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:20:06.975370 3736 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:20:06.975296 3738 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:20:06.975615 3730 server_base.cc:1034] running on GCE node
I20260929 03:20:06.975845 3730 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:20:06.976109 3730 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:20:06.977272 3730 hybrid_clock.cc:648] HybridClock initialized: now 1790652006977260 us; error 30 us; skew 500 ppm
Sep 29 03:20:06 dist-test-slave-c9qh krb5kdc[2425](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652006, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:20:06.980015 3730 init.cc:377] Logged in from keytab as kudu/127.20.178.66@KRBTEST.COM (short username kudu)
I20260929 03:20:06.980438 3730 webserver.cc:460] Webserver started at http://127.20.178.66:36125/ using document root <none> and password file <none>
I20260929 03:20:06.980734 3730 fs_manager.cc:359] Metadata directory not provided
I20260929 03:20:06.980803 3730 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:20:06.982218 3730 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 03:20:06.982868 3745 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:20:06.983079 3730 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
W20260929 03:20:06.983366 3727 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:20:06.876744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:33493 (local address 127.20.178.65:59181)
0929 03:20:06.876935 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0929 03:20:06.877009 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0929 03:20:06.877427 (+ 418us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:20:06.983182 (+105755us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.178.126:33493: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":106,"threads_started":1}
I20260929 03:20:06.983172 3730 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/minicluster-data/ts-1/data,/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/minicluster-data/ts-1/wal
uuid: "c4d16a71b9b44e9f95766d5ed5ede8b9"
format_stamp: "Formatted at 2026-09-29 03:20:05 on dist-test-slave-c9qh"
server_key: "53dbba98e9bfa97db74cadca73323601"
server_key_iv: "696b9603205ac7a46e23303ea2764a6f"
server_key_version: "encryptionkey@0"
I20260929 03:20:06.983574 3730 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
W20260929 03:20:06.983721 3725 heartbeater.cc:644] Failed to heartbeat to 127.20.178.126:33493 (0 consecutive failures): Network error: Failed to ping master at 127.20.178.126:33493: Client connection negotiation failed: client connection to 127.20.178.126:33493: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260929 03:20:06.984033 3727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:20:06.983920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:33493 (local address 127.20.178.65:53285)
0929 03:20:06.983973 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0929 03:20:06.984009 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.178.126:33493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260929 03:20:06.984360 3727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:20:06.984267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:33493 (local address 127.20.178.65:46583)
0929 03:20:06.984317 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0929 03:20:06.984342 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.178.126:33493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260929 03:20:06.984504 3725 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260929 03:20:06.991439 3730 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:20:06.993667 3730 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:20:06.993809 3730 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:20:06.994056 3730 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:20:06.994350 3730 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:20:06.994408 3730 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:20:06.994447 3730 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:20:06.994477 3730 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:20:07.000295 3730 rpc_server.cc:307] RPC server started. Bound to: 127.20.178.66:37709
I20260929 03:20:07.000370 3858 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.178.66:37709 every 8 connection(s)
I20260929 03:20:07.000759 3730 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/minicluster-data/ts-1/data/info.pb
I20260929 03:20:07.001330 21193 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuoaHbX/build/release/bin/kudu as pid 3730
I20260929 03:20:07.001617 21193 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuoaHbX/build/release/bin/kudu
/tmp/dist-test-taskuoaHbX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.178.67:37489
--local_ip_for_outbound_sockets=127.20.178.67
--tserver_master_addrs=127.20.178.126:33493
--webserver_port=34703
--webserver_interface=127.20.178.67
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.67.keytab
--principal=kudu/127.20.178.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.178.84:42811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 03:20:07.001906 3861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:20:07.001446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:33493 (local address 127.20.178.66:55495)
0929 03:20:07.001620 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0929 03:20:07.001728 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.178.126:33493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":85,"threads_started":1}
W20260929 03:20:07.002293 3859 heartbeater.cc:644] Failed to heartbeat to 127.20.178.126:33493 (0 consecutive failures): Network error: Failed to ping master at 127.20.178.126:33493: Client connection negotiation failed: client connection to 127.20.178.126:33493: connect: Connection refused (error 111)
I20260929 03:20:07.002594 3861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:20:07.002474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:33493 (local address 127.20.178.66:40301)
0929 03:20:07.002529 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0929 03:20:07.002561 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.178.126:33493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260929 03:20:07.002902 3861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:20:07.002806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:33493 (local address 127.20.178.66:59611)
0929 03:20:07.002847 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0929 03:20:07.002875 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.178.126:33493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260929 03:20:07.003041 3859 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 03:20:07.081890 3862 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:20:07.082080 3862 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:20:07.082103 3862 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:20:07.083222 3862 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:20:07.083261 3862 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:20:07.083310 3862 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.178.67
I20260929 03:20:07.084806 3862 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.178.84:42811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.67.keytab
--principal=kudu/127.20.178.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.178.67:37489
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.178.67
--webserver_port=34703
--tserver_master_addrs=127.20.178.126:33493
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3862
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.178.67
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:20:07.085038 3862 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:20:07.085251 3862 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:20:07.090252 3867 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:20:07.090245 3870 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:20:07.090381 3862 server_base.cc:1034] running on GCE node
W20260929 03:20:07.090245 3868 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:20:07.090634 3862 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:20:07.090868 3862 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:20:07.092028 3862 hybrid_clock.cc:648] HybridClock initialized: now 1790652007092017 us; error 43 us; skew 500 ppm
Sep 29 03:20:07 dist-test-slave-c9qh krb5kdc[2425](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652007, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:20:07.094496 3862 init.cc:377] Logged in from keytab as kudu/127.20.178.67@KRBTEST.COM (short username kudu)
I20260929 03:20:07.094910 3862 webserver.cc:460] Webserver started at http://127.20.178.67:34703/ using document root <none> and password file <none>
I20260929 03:20:07.095136 3862 fs_manager.cc:359] Metadata directory not provided
I20260929 03:20:07.095207 3862 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:20:07.096554 3862 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260929 03:20:07.097218 3877 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:20:07.097376 3862 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260929 03:20:07.097474 3862 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/minicluster-data/ts-2/data,/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/minicluster-data/ts-2/wal
uuid: "0fdb3aa2f42046dda60d96844a22c272"
format_stamp: "Formatted at 2026-09-29 03:20:05 on dist-test-slave-c9qh"
server_key: "5e809bd46f719d2e6cf2f84c6b87e4d6"
server_key_iv: "2d588bc3d30278eb60f5ddc3fa8b86c0"
server_key_version: "encryptionkey@0"
I20260929 03:20:07.097821 3862 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:20:07.110554 3862 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:20:07.112269 3862 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:20:07.112432 3862 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:20:07.112722 3862 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:20:07.113034 3862 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:20:07.113089 3862 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:20:07.113130 3862 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:20:07.113157 3862 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:20:07.118848 3862 rpc_server.cc:307] RPC server started. Bound to: 127.20.178.67:37489
I20260929 03:20:07.118892 3990 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.178.67:37489 every 8 connection(s)
I20260929 03:20:07.119185 3862 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651852476208-21193-0/minicluster-data/ts-2/data/info.pb
I20260929 03:20:07.120082 3993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:20:07.119749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:33493 (local address 127.20.178.67:49613)
0929 03:20:07.119882 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0929 03:20:07.119949 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.178.126:33493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":62,"threads_started":1}
W20260929 03:20:07.120491 3991 heartbeater.cc:644] Failed to heartbeat to 127.20.178.126:33493 (0 consecutive failures): Network error: Failed to ping master at 127.20.178.126:33493: Client connection negotiation failed: client connection to 127.20.178.126:33493: connect: Connection refused (error 111)
I20260929 03:20:07.120808 3993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:20:07.120718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:33493 (local address 127.20.178.67:39505)
0929 03:20:07.120766 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0929 03:20:07.120791 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.178.126:33493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260929 03:20:07.121137 3993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:20:07.121046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:33493 (local address 127.20.178.67:36187)
0929 03:20:07.121091 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0929 03:20:07.121124 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.178.126:33493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260929 03:20:07.121258 3991 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260929 03:20:07.126793 21193 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuoaHbX/build/release/bin/kudu as pid 3862
I20260929 03:20:07.985154 3995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:20:07.984891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:33493 (local address 127.20.178.65:38511)
0929 03:20:07.985040 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0929 03:20:07.985091 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.178.126:33493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":62,"threads_started":1}
I20260929 03:20:08.003573 3996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:20:08.003370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:33493 (local address 127.20.178.66:56223)
0929 03:20:08.003489 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0929 03:20:08.003524 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.178.126:33493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":63,"threads_started":1}
I20260929 03:20:08.121884 3997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:20:08.121636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:33493 (local address 127.20.178.67:33911)
0929 03:20:08.121764 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0929 03:20:08.121793 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.178.126:33493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":67,"threads_started":1}
I20260929 03:20:08.234771 3998 raft_consensus.cc:482] T 13a38243527941ff9914cdf90b184502 P 5071e4d22012499c9e198b9b39515856 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:20:08.234890 3998 raft_consensus.cc:504] T 13a38243527941ff9914cdf90b184502 P 5071e4d22012499c9e198b9b39515856 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4d16a71b9b44e9f95766d5ed5ede8b9" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 37709 } } peers { permanent_uuid: "0fdb3aa2f42046dda60d96844a22c272" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 37489 } } peers { permanent_uuid: "5071e4d22012499c9e198b9b39515856" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 44525 } }
I20260929 03:20:08.235210 3998 leader_election.cc:288] T 13a38243527941ff9914cdf90b184502 P 5071e4d22012499c9e198b9b39515856 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c4d16a71b9b44e9f95766d5ed5ede8b9 (127.20.178.66:37709), 0fdb3aa2f42046dda60d96844a22c272 (127.20.178.67:37489)
Sep 29 03:20:08 dist-test-slave-c9qh krb5kdc[2425](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652006, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.65@KRBTEST.COM for kudu/127.20.178.67@KRBTEST.COM
Sep 29 03:20:08 dist-test-slave-c9qh krb5kdc[2425](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652006, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.65@KRBTEST.COM for kudu/127.20.178.66@KRBTEST.COM
I20260929 03:20:08.247584 3999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:20:08.235434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.67:37489 (local address 127.20.178.65:48043)
0929 03:20:08.235531 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0929 03:20:08.235542 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 03:20:08.235601 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:20:08.236592 (+ 991us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:20:08.236595 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:20:08.237169 (+ 574us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:20:08.237439 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:20:08.237447 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:20:08.239076 (+ 1629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:20:08.239077 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:20:08.240425 (+ 1348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:20:08.240428 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:20:08.241264 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:20:08.241264 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:20:08.241411 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:20:08.242028 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:20:08.242039 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:20:08.243878 (+ 1839us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:20:08.245865 (+ 1987us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:20:08.245871 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:20:08.245873 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:20:08.246161 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:20:08.246393 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:20:08.246393 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:20:08.246394 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:20:08.246543 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:20:08.247011 (+ 468us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:20:08.247016 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:20:08.247176 (+ 160us) client_negotiation.cc:769] Sending connection context
0929 03:20:08.247291 (+ 115us) client_negotiation.cc:240] Negotiation successful
0929 03:20:08.247451 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":86,"mutex_wait_us":93,"thread_start_us":50,"threads_started":1}
I20260929 03:20:08.247628 4001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:20:08.235495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.65:48043 (local address 127.20.178.67:37489)
0929 03:20:08.235712 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0929 03:20:08.235720 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 03:20:08.235748 (+ 28us) server_negotiation.cc:407] Connection header received
0929 03:20:08.236305 (+ 557us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:20:08.236308 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:20:08.236366 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:20:08.236530 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:20:08.237568 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:20:08.238976 (+ 1408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:20:08.240514 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:20:08.241167 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:20:08.241289 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:20:08.243988 (+ 2699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:20:08.243992 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:20:08.243992 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:20:08.244031 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:20:08.245783 (+ 1752us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:20:08.246242 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:20:08.246245 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:20:08.246245 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:20:08.246326 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:20:08.246612 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:20:08.246613 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:20:08.246613 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:20:08.246913 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:20:08.247037 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0929 03:20:08.247344 (+ 307us) server_negotiation.cc:299] Negotiation successful
0929 03:20:08.247506 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":94,"threads_started":1}
I20260929 03:20:08.247891 3995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:20:08.235377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.66:37709 (local address 127.20.178.65:44027)
0929 03:20:08.235438 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0929 03:20:08.235449 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 03:20:08.235684 (+ 235us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:20:08.236592 (+ 908us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:20:08.236595 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:20:08.237167 (+ 572us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:20:08.237439 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:20:08.237447 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:20:08.239079 (+ 1632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:20:08.239080 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:20:08.240425 (+ 1345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:20:08.240428 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:20:08.241262 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:20:08.241264 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:20:08.241411 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:20:08.242028 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:20:08.242039 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:20:08.244288 (+ 2249us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:20:08.246521 (+ 2233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:20:08.246522 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:20:08.246522 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:20:08.246737 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:20:08.246960 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:20:08.246961 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:20:08.246961 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:20:08.247025 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:20:08.247490 (+ 465us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:20:08.247490 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:20:08.247528 (+ 38us) client_negotiation.cc:769] Sending connection context
0929 03:20:08.247725 (+ 197us) client_negotiation.cc:240] Negotiation successful
0929 03:20:08.247833 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43,"mutex_wait_us":82}
I20260929 03:20:08.248250 4000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:20:08.235493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.65:44027 (local address 127.20.178.66:37709)
0929 03:20:08.235655 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0929 03:20:08.235668 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 03:20:08.235706 (+ 38us) server_negotiation.cc:407] Connection header received
0929 03:20:08.236305 (+ 599us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:20:08.236308 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:20:08.236366 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:20:08.236521 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:20:08.237568 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:20:08.238969 (+ 1401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:20:08.240514 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:20:08.241167 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:20:08.241289 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:20:08.244414 (+ 3125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:20:08.244417 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:20:08.244418 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:20:08.244429 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:20:08.246432 (+ 2003us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:20:08.246815 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:20:08.246817 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:20:08.246817 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:20:08.246893 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:20:08.247090 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:20:08.247091 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:20:08.247091 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:20:08.247389 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:20:08.247461 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0929 03:20:08.248066 (+ 605us) server_negotiation.cc:299] Negotiation successful
0929 03:20:08.248176 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":103,"threads_started":1}
W20260929 03:20:08.248955 3612 leader_election.cc:341] T 13a38243527941ff9914cdf90b184502 P 5071e4d22012499c9e198b9b39515856 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0fdb3aa2f42046dda60d96844a22c272 (127.20.178.67:37489): Not found: Tablet not found: 13a38243527941ff9914cdf90b184502
W20260929 03:20:08.249051 3611 leader_election.cc:341] T 13a38243527941ff9914cdf90b184502 P 5071e4d22012499c9e198b9b39515856 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c4d16a71b9b44e9f95766d5ed5ede8b9 (127.20.178.66:37709): Not found: Tablet not found: 13a38243527941ff9914cdf90b184502
I20260929 03:20:08.249099 3611 leader_election.cc:302] T 13a38243527941ff9914cdf90b184502 P 5071e4d22012499c9e198b9b39515856 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5071e4d22012499c9e198b9b39515856; no voters: 0fdb3aa2f42046dda60d96844a22c272, c4d16a71b9b44e9f95766d5ed5ede8b9
I20260929 03:20:08.249202 3998 raft_consensus.cc:2726] T 13a38243527941ff9914cdf90b184502 P 5071e4d22012499c9e198b9b39515856 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 03:20:08.985836 4002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:20:08.985621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:33493 (local address 127.20.178.65:36007)
0929 03:20:08.985748 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0929 03:20:08.985791 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.178.126:33493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":81,"threads_started":1}
I20260929 03:20:09.004345 4003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:20:09.004112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:33493 (local address 127.20.178.66:49249)
0929 03:20:09.004264 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0929 03:20:09.004305 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.178.126:33493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":78,"threads_started":1}
I20260929 03:20:09.122571 4004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:20:09.122381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:33493 (local address 127.20.178.67:41497)
0929 03:20:09.122503 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0929 03:20:09.122535 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.178.126:33493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":81,"threads_started":1}
I20260929 03:20:09.986584 4005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:20:09.986357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:33493 (local address 127.20.178.65:55271)
0929 03:20:09.986489 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0929 03:20:09.986537 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.178.126:33493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":64,"threads_started":1}
I20260929 03:20:10.005080 4006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:20:10.004876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:33493 (local address 127.20.178.66:59687)
0929 03:20:10.004998 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0929 03:20:10.005040 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.178.126:33493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":58,"threads_started":1}
I20260929 03:20:10.123411 4007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:20:10.123186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:33493 (local address 127.20.178.67:34607)
0929 03:20:10.123322 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0929 03:20:10.123367 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.178.126:33493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":60,"threads_started":1}
I20260929 03:20:10.246773 4008 raft_consensus.cc:482] T 13a38243527941ff9914cdf90b184502 P 5071e4d22012499c9e198b9b39515856 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:20:10.246896 4008 raft_consensus.cc:504] T 13a38243527941ff9914cdf90b184502 P 5071e4d22012499c9e198b9b39515856 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4d16a71b9b44e9f95766d5ed5ede8b9" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 37709 } } peers { permanent_uuid: "0fdb3aa2f42046dda60d96844a22c272" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 37489 } } peers { permanent_uuid: "5071e4d22012499c9e198b9b39515856" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 44525 } }
I20260929 03:20:10.247131 4008 leader_election.cc:288] T 13a38243527941ff9914cdf90b184502 P 5071e4d22012499c9e198b9b39515856 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c4d16a71b9b44e9f95766d5ed5ede8b9 (127.20.178.66:37709), 0fdb3aa2f42046dda60d96844a22c272 (127.20.178.67:37489)
W20260929 03:20:10.247547 3611 leader_election.cc:341] T 13a38243527941ff9914cdf90b184502 P 5071e4d22012499c9e198b9b39515856 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c4d16a71b9b44e9f95766d5ed5ede8b9 (127.20.178.66:37709): Not found: Tablet not found: 13a38243527941ff9914cdf90b184502
W20260929 03:20:10.247674 3612 leader_election.cc:341] T 13a38243527941ff9914cdf90b184502 P 5071e4d22012499c9e198b9b39515856 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0fdb3aa2f42046dda60d96844a22c272 (127.20.178.67:37489): Not found: Tablet not found: 13a38243527941ff9914cdf90b184502
I20260929 03:20:10.247714 3612 leader_election.cc:302] T 13a38243527941ff9914cdf90b184502 P 5071e4d22012499c9e198b9b39515856 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5071e4d22012499c9e198b9b39515856; no voters: 0fdb3aa2f42046dda60d96844a22c272, c4d16a71b9b44e9f95766d5ed5ede8b9
I20260929 03:20:10.247818 4008 raft_consensus.cc:2726] T 13a38243527941ff9914cdf90b184502 P 5071e4d22012499c9e198b9b39515856 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 03:20:10.987423 4009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:20:10.987148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:33493 (local address 127.20.178.65:55119)
0929 03:20:10.987331 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0929 03:20:10.987376 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.178.126:33493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":97,"threads_started":1}
I20260929 03:20:11.005738 4010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:20:11.005528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:33493 (local address 127.20.178.66:50409)
0929 03:20:11.005641 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0929 03:20:11.005684 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.178.126:33493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":52,"threads_started":1}
I20260929 03:20:11.124384 4011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:20:11.124077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:33493 (local address 127.20.178.67:38495)
0929 03:20:11.124296 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0929 03:20:11.124342 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.178.126:33493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":56,"threads_started":1}
I20260929 03:20:11.988121 4012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:20:11.987901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:33493 (local address 127.20.178.65:39555)
0929 03:20:11.988030 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0929 03:20:11.988066 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.178.126:33493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":71,"threads_started":1}
I20260929 03:20:12.006444 4013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:20:12.006234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:33493 (local address 127.20.178.66:56239)
0929 03:20:12.006367 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0929 03:20:12.006406 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.178.126:33493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":92,"threads_started":1}
I20260929 03:20:12.125074 4014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:20:12.124881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:33493 (local address 127.20.178.67:33069)
0929 03:20:12.125006 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0929 03:20:12.125038 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.178.126:33493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":70,"threads_started":1}
I20260929 03:20:12.988852 4015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:20:12.988565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:33493 (local address 127.20.178.65:57645)
0929 03:20:12.988734 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0929 03:20:12.988780 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.178.126:33493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":100,"threads_started":1}
I20260929 03:20:13.007054 4016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:20:13.006855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:33493 (local address 127.20.178.66:49825)
0929 03:20:13.006977 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0929 03:20:13.007013 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.178.126:33493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":56,"threads_started":1}
I20260929 03:20:13.028818 4017 raft_consensus.cc:482] T 13a38243527941ff9914cdf90b184502 P 5071e4d22012499c9e198b9b39515856 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:20:13.028913 4017 raft_consensus.cc:504] T 13a38243527941ff9914cdf90b184502 P 5071e4d22012499c9e198b9b39515856 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4d16a71b9b44e9f95766d5ed5ede8b9" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 37709 } } peers { permanent_uuid: "0fdb3aa2f42046dda60d96844a22c272" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 37489 } } peers { permanent_uuid: "5071e4d22012499c9e198b9b39515856" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 44525 } }
I20260929 03:20:13.029130 4017 leader_election.cc:288] T 13a38243527941ff9914cdf90b184502 P 5071e4d22012499c9e198b9b39515856 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c4d16a71b9b44e9f95766d5ed5ede8b9 (127.20.178.66:37709), 0fdb3aa2f42046dda60d96844a22c272 (127.20.178.67:37489)
W20260929 03:20:13.029515 3612 leader_election.cc:341] T 13a38243527941ff9914cdf90b184502 P 5071e4d22012499c9e198b9b39515856 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0fdb3aa2f42046dda60d96844a22c272 (127.20.178.67:37489): Not found: Tablet not found: 13a38243527941ff9914cdf90b184502
W20260929 03:20:13.029644 3611 leader_election.cc:341] T 13a38243527941ff9914cdf90b184502 P 5071e4d22012499c9e198b9b39515856 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c4d16a71b9b44e9f95766d5ed5ede8b9 (127.20.178.66:37709): Not found: Tablet not found: 13a38243527941ff9914cdf90b184502
I20260929 03:20:13.029709 3611 leader_election.cc:302] T 13a38243527941ff9914cdf90b184502 P 5071e4d22012499c9e198b9b39515856 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5071e4d22012499c9e198b9b39515856; no voters: 0fdb3aa2f42046dda60d96844a22c272, c4d16a71b9b44e9f95766d5ed5ede8b9
I20260929 03:20:13.029810 4017 raft_consensus.cc:2726] T 13a38243527941ff9914cdf90b184502 P 5071e4d22012499c9e198b9b39515856 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 03:20:13.125705 4018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:20:13.125505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:33493 (local address 127.20.178.67:48825)
0929 03:20:13.125625 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0929 03:20:13.125663 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.178.126:33493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":75,"threads_started":1}
I20260929 03:20:13.989508 4019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:20:13.989271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:33493 (local address 127.20.178.65:57761)
0929 03:20:13.989428 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0929 03:20:13.989464 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.178.126:33493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":60,"threads_started":1}
I20260929 03:20:14.007673 4020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:20:14.007482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:33493 (local address 127.20.178.66:38235)
0929 03:20:14.007604 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0929 03:20:14.007634 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.178.126:33493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":75,"threads_started":1}
I20260929 03:20:14.126457 4021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:20:14.126236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:33493 (local address 127.20.178.67:59101)
0929 03:20:14.126382 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0929 03:20:14.126416 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.178.126:33493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":71,"threads_started":1}
I20260929 03:20:14.990213 4022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:20:14.989998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:33493 (local address 127.20.178.65:53083)
0929 03:20:14.990126 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0929 03:20:14.990166 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.178.126:33493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":62,"threads_started":1}
I20260929 03:20:15.008287 4023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:20:15.008103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:33493 (local address 127.20.178.66:58145)
0929 03:20:15.008216 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0929 03:20:15.008245 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.178.126:33493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":55,"threads_started":1}
I20260929 03:20:15.127218 4024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:20:15.127001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:33493 (local address 127.20.178.67:39865)
0929 03:20:15.127129 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0929 03:20:15.127170 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.178.126:33493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":61,"threads_started":1}
I20260929 03:20:15.344357 4025 raft_consensus.cc:482] T 13a38243527941ff9914cdf90b184502 P 5071e4d22012499c9e198b9b39515856 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:20:15.344465 4025 raft_consensus.cc:504] T 13a38243527941ff9914cdf90b184502 P 5071e4d22012499c9e198b9b39515856 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4d16a71b9b44e9f95766d5ed5ede8b9" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 37709 } } peers { permanent_uuid: "0fdb3aa2f42046dda60d96844a22c272" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 37489 } } peers { permanent_uuid: "5071e4d22012499c9e198b9b39515856" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 44525 } }
I20260929 03:20:15.344689 4025 leader_election.cc:288] T 13a38243527941ff9914cdf90b184502 P 5071e4d22012499c9e198b9b39515856 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c4d16a71b9b44e9f95766d5ed5ede8b9 (127.20.178.66:37709), 0fdb3aa2f42046dda60d96844a22c272 (127.20.178.67:37489)
W20260929 03:20:15.345041 3611 leader_election.cc:341] T 13a38243527941ff9914cdf90b184502 P 5071e4d22012499c9e198b9b39515856 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c4d16a71b9b44e9f95766d5ed5ede8b9 (127.20.178.66:37709): Not found: Tablet not found: 13a38243527941ff9914cdf90b184502
W20260929 03:20:15.345129 3612 leader_election.cc:341] T 13a38243527941ff9914cdf90b184502 P 5071e4d22012499c9e198b9b39515856 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0fdb3aa2f42046dda60d96844a22c272 (127.20.178.67:37489): Not found: Tablet not found: 13a38243527941ff9914cdf90b184502
I20260929 03:20:15.345209 3612 leader_election.cc:302] T 13a38243527941ff9914cdf90b184502 P 5071e4d22012499c9e198b9b39515856 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5071e4d22012499c9e198b9b39515856; no voters: 0fdb3aa2f42046dda60d96844a22c272, c4d16a71b9b44e9f95766d5ed5ede8b9
I20260929 03:20:15.345304 4025 raft_consensus.cc:2726] T 13a38243527941ff9914cdf90b184502 P 5071e4d22012499c9e198b9b39515856 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 03:20:15.991008 4026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:20:15.990796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:33493 (local address 127.20.178.65:51163)
0929 03:20:15.990920 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0929 03:20:15.990961 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.178.126:33493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":56,"threads_started":1}
I20260929 03:20:16.008960 4027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:20:16.008775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:33493 (local address 127.20.178.66:36333)
0929 03:20:16.008882 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0929 03:20:16.008917 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.178.126:33493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":54,"threads_started":1}
I20260929 03:20:16.127914 4028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:20:16.127728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:33493 (local address 127.20.178.67:43663)
0929 03:20:16.127837 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0929 03:20:16.127871 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.178.126:33493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":57,"threads_started":1}
I20260929 03:20:16.991655 4029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:20:16.991447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:33493 (local address 127.20.178.65:39323)
0929 03:20:16.991560 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0929 03:20:16.991610 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.178.126:33493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":61,"threads_started":1}
I20260929 03:20:17.009621 4030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:20:17.009411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:33493 (local address 127.20.178.66:41003)
0929 03:20:17.009543 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0929 03:20:17.009580 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.178.126:33493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":60,"threads_started":1}
I20260929 03:20:17.128582 4031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:20:17.128370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:33493 (local address 127.20.178.67:47279)
0929 03:20:17.128499 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0929 03:20:17.128537 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.178.126:33493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":59,"threads_started":1}
I20260929 03:20:17.591749 4032 raft_consensus.cc:482] T 13a38243527941ff9914cdf90b184502 P 5071e4d22012499c9e198b9b39515856 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:20:17.591866 4032 raft_consensus.cc:504] T 13a38243527941ff9914cdf90b184502 P 5071e4d22012499c9e198b9b39515856 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4d16a71b9b44e9f95766d5ed5ede8b9" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 37709 } } peers { permanent_uuid: "0fdb3aa2f42046dda60d96844a22c272" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 37489 } } peers { permanent_uuid: "5071e4d22012499c9e198b9b39515856" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 44525 } }
I20260929 03:20:17.592063 4032 leader_election.cc:288] T 13a38243527941ff9914cdf90b184502 P 5071e4d22012499c9e198b9b39515856 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c4d16a71b9b44e9f95766d5ed5ede8b9 (127.20.178.66:37709), 0fdb3aa2f42046dda60d96844a22c272 (127.20.178.67:37489)
W20260929 03:20:17.592372 3611 leader_election.cc:341] T 13a38243527941ff9914cdf90b184502 P 5071e4d22012499c9e198b9b39515856 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c4d16a71b9b44e9f95766d5ed5ede8b9 (127.20.178.66:37709): Not found: Tablet not found: 13a38243527941ff9914cdf90b184502
W20260929 03:20:17.592446 3612 leader_election.cc:341] T 13a38243527941ff9914cdf90b184502 P 5071e4d22012499c9e198b9b39515856 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0fdb3aa2f42046dda60d96844a22c272 (127.20.178.67:37489): Not found: Tablet not found: 13a38243527941ff9914cdf90b184502
I20260929 03:20:17.592543 3612 leader_election.cc:302] T 13a38243527941ff9914cdf90b184502 P 5071e4d22012499c9e198b9b39515856 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5071e4d22012499c9e198b9b39515856; no voters: 0fdb3aa2f42046dda60d96844a22c272, c4d16a71b9b44e9f95766d5ed5ede8b9
I20260929 03:20:17.592649 4032 raft_consensus.cc:2726] T 13a38243527941ff9914cdf90b184502 P 5071e4d22012499c9e198b9b39515856 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 03:20:17.992262 4033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:20:17.992069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:33493 (local address 127.20.178.65:39391)
0929 03:20:17.992182 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0929 03:20:17.992216 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.178.126:33493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":60,"threads_started":1}
I20260929 03:20:18.010269 4034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:20:18.010049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:33493 (local address 127.20.178.66:45057)
0929 03:20:18.010200 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0929 03:20:18.010229 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.178.126:33493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":98,"threads_started":1}
I20260929 03:20:18.129328 4035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:20:18.129140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:33493 (local address 127.20.178.67:45101)
0929 03:20:18.129253 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0929 03:20:18.129285 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.178.126:33493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":62,"threads_started":1}
I20260929 03:20:18.993084 4036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:20:18.992859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:33493 (local address 127.20.178.65:35563)
0929 03:20:18.993003 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0929 03:20:18.993039 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.178.126:33493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":90,"threads_started":1}
I20260929 03:20:19.011022 4037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:20:19.010808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:33493 (local address 127.20.178.66:54485)
0929 03:20:19.010940 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0929 03:20:19.010978 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.178.126:33493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":63,"threads_started":1}
I20260929 03:20:19.130002 4038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:20:19.129736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:33493 (local address 127.20.178.67:56205)
0929 03:20:19.129904 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0929 03:20:19.129944 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.178.126:33493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":78,"threads_started":1}
I20260929 03:20:19.993716 4039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:20:19.993507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:33493 (local address 127.20.178.65:52763)
0929 03:20:19.993636 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0929 03:20:19.993672 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.178.126:33493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":59,"threads_started":1}
I20260929 03:20:20.011675 4040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:20:20.011484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:33493 (local address 127.20.178.66:42407)
0929 03:20:20.011603 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0929 03:20:20.011636 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.178.126:33493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":55,"threads_started":1}
I20260929 03:20:20.130767 4041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:20:20.130544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:33493 (local address 127.20.178.67:53043)
0929 03:20:20.130684 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0929 03:20:20.130723 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.178.126:33493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":57,"threads_started":1}
I20260929 03:20:20.994323 4042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:20:20.994127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:33493 (local address 127.20.178.65:60983)
0929 03:20:20.994244 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0929 03:20:20.994276 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.178.126:33493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":62,"threads_started":1}
I20260929 03:20:21.012295 4043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:20:21.012083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:33493 (local address 127.20.178.66:38955)
0929 03:20:21.012219 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0929 03:20:21.012251 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.178.126:33493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":82,"threads_started":1}
I20260929 03:20:21.131510 4044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:20:21.131288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:33493 (local address 127.20.178.67:53251)
0929 03:20:21.131428 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0929 03:20:21.131464 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.178.126:33493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":74,"threads_started":1}
I20260929 03:20:21.995050 4045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:20:21.994847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:33493 (local address 127.20.178.65:37163)
0929 03:20:21.994976 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0929 03:20:21.995009 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.178.126:33493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":90,"threads_started":1}
I20260929 03:20:22.013078 4046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:20:22.012835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:33493 (local address 127.20.178.66:41721)
0929 03:20:22.012998 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0929 03:20:22.013035 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.178.126:33493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":72,"threads_started":1}
I20260929 03:20:22.130376 21193 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuoaHbX/build/release/bin/kudu with pid 3566
I20260929 03:20:22.132156 4047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:20:22.131965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:33493 (local address 127.20.178.67:36579)
0929 03:20:22.132087 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0929 03:20:22.132122 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.178.126:33493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":74,"threads_started":1}
I20260929 03:20:22.135556 21193 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuoaHbX/build/release/bin/kudu with pid 3730
I20260929 03:20:22.140228 21193 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuoaHbX/build/release/bin/kudu with pid 3862
2026-09-29T03:20:22Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (18159 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:20:22 dist-test-slave-c9qh krb5kdc[4050](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:20:22 dist-test-slave-c9qh krb5kdc[4050](info): set up 2 sockets
Sep 29 03:20:22 dist-test-slave-c9qh krb5kdc[4050](info): commencing operation
krb5kdc: starting...
W20260929 03:20:23.172981 21193 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/krb5kdc/test-user.keytab.
Sep 29 03:20:23 dist-test-slave-c9qh krb5kdc[4050](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652023, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:20:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:20:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.178.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.178.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.178.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.126.keytab.
Entry for principal kudu/127.20.178.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.126.keytab.
I20260929 03:20:23.287531 21193 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuoaHbX/build/release/bin/kudu
/tmp/dist-test-taskuoaHbX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.178.126:36419
--webserver_interface=127.20.178.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.126.keytab
--principal=kudu/127.20.178.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.178.84:42735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.178.126:36419
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:20:23.367832 4063 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:20:23.368032 4063 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:20:23.368054 4063 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:20:23.369159 4063 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260929 03:20:23.369197 4063 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260929 03:20:23.369210 4063 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260929 03:20:23.369222 4063 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:20:23.369235 4063 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:20:23.369248 4063 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:20:23.369266 4063 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:20:23.369279 4063 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:20:23.370656 4063 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.178.84:42735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.20.178.126:36419
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.126.keytab
--principal=kudu/127.20.178.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.178.126:36419
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.178.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4063
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:20:23.370836 4063 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:20:23.371006 4063 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 03:20:23.376093 4063 server_base.cc:1034] running on GCE node
W20260929 03:20:23.376065 4069 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:20:23.376073 4071 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:20:23.376066 4068 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:20:23.376441 4063 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:20:23.376773 4063 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:20:23.377967 4063 hybrid_clock.cc:648] HybridClock initialized: now 1790652023377957 us; error 36 us; skew 500 ppm
Sep 29 03:20:23 dist-test-slave-c9qh krb5kdc[4050](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652023, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:20:23.387219 4063 init.cc:377] Logged in from keytab as kudu/127.20.178.126@KRBTEST.COM (short username kudu)
I20260929 03:20:23.387706 4063 webserver.cc:460] Webserver started at http://127.20.178.126:38909/ using document root <none> and password file <none>
I20260929 03:20:23.387924 4063 fs_manager.cc:359] Metadata directory not provided
I20260929 03:20:23.387991 4063 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:20:23.388129 4063 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:20:23.389204 4063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/minicluster-data/master-0/data/instance:
uuid: "31cae0b535f04bca833ae9ab7318f4b2"
format_stamp: "Formatted at 2026-09-29 03:20:23 on dist-test-slave-c9qh"
server_key: "986112c2b7206bc8770ea147775edf07"
server_key_iv: "c69716b22cda7a2a894ffe04d5ff4e22"
server_key_version: "encryptionkey@0"
I20260929 03:20:23.389554 4063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/minicluster-data/master-0/wal/instance:
uuid: "31cae0b535f04bca833ae9ab7318f4b2"
format_stamp: "Formatted at 2026-09-29 03:20:23 on dist-test-slave-c9qh"
server_key: "986112c2b7206bc8770ea147775edf07"
server_key_iv: "c69716b22cda7a2a894ffe04d5ff4e22"
server_key_version: "encryptionkey@0"
I20260929 03:20:23.391227 4063 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260929 03:20:23.392036 4078 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:20:23.392202 4063 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 03:20:23.392268 4063 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/minicluster-data/master-0/data,/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/minicluster-data/master-0/wal
uuid: "31cae0b535f04bca833ae9ab7318f4b2"
format_stamp: "Formatted at 2026-09-29 03:20:23 on dist-test-slave-c9qh"
server_key: "986112c2b7206bc8770ea147775edf07"
server_key_iv: "c69716b22cda7a2a894ffe04d5ff4e22"
server_key_version: "encryptionkey@0"
I20260929 03:20:23.392318 4063 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:20:23.402540 4063 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:20:23.404289 4063 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:20:23.404390 4063 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:20:23.408051 4063 rpc_server.cc:307] RPC server started. Bound to: 127.20.178.126:36419
I20260929 03:20:23.408110 4130 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.178.126:36419 every 8 connection(s)
I20260929 03:20:23.408452 4063 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/minicluster-data/master-0/data/info.pb
I20260929 03:20:23.409305 4131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:20:23.411455 4131 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31cae0b535f04bca833ae9ab7318f4b2: Bootstrap starting.
I20260929 03:20:23.411819 21193 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuoaHbX/build/release/bin/kudu as pid 4063
I20260929 03:20:23.411892 21193 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/minicluster-data/master-0/wal/instance
I20260929 03:20:23.412071 4131 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 31cae0b535f04bca833ae9ab7318f4b2: Neither blocks nor log segments found. Creating new log.
I20260929 03:20:23.412093 21193 external_mini_cluster.cc:1428] Setting key b24b38e89d0a41e25d248b6d5d74f52d
I20260929 03:20:23.412407 4131 log.cc:824] T 00000000000000000000000000000000 P 31cae0b535f04bca833ae9ab7318f4b2: Log is configured to *not* fsync() on all Append() calls
I20260929 03:20:23.413342 4131 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31cae0b535f04bca833ae9ab7318f4b2: No bootstrap required, opened a new log
I20260929 03:20:23.415602 4131 raft_consensus.cc:348] T 00000000000000000000000000000000 P 31cae0b535f04bca833ae9ab7318f4b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31cae0b535f04bca833ae9ab7318f4b2" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 36419 } }
I20260929 03:20:23.415778 4131 raft_consensus.cc:374] T 00000000000000000000000000000000 P 31cae0b535f04bca833ae9ab7318f4b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:20:23.415827 4131 raft_consensus.cc:729] T 00000000000000000000000000000000 P 31cae0b535f04bca833ae9ab7318f4b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31cae0b535f04bca833ae9ab7318f4b2, State: Initialized, Role: FOLLOWER
I20260929 03:20:23.415947 4131 consensus_queue.cc:260] T 00000000000000000000000000000000 P 31cae0b535f04bca833ae9ab7318f4b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31cae0b535f04bca833ae9ab7318f4b2" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 36419 } }
I20260929 03:20:23.416029 4131 raft_consensus.cc:388] T 00000000000000000000000000000000 P 31cae0b535f04bca833ae9ab7318f4b2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:20:23.416071 4131 raft_consensus.cc:482] T 00000000000000000000000000000000 P 31cae0b535f04bca833ae9ab7318f4b2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:20:23.416119 4131 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 31cae0b535f04bca833ae9ab7318f4b2 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:20:23.416711 4131 raft_consensus.cc:504] T 00000000000000000000000000000000 P 31cae0b535f04bca833ae9ab7318f4b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31cae0b535f04bca833ae9ab7318f4b2" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 36419 } }
I20260929 03:20:23.416811 4131 leader_election.cc:302] T 00000000000000000000000000000000 P 31cae0b535f04bca833ae9ab7318f4b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 31cae0b535f04bca833ae9ab7318f4b2; no voters:
I20260929 03:20:23.416976 4131 leader_election.cc:288] T 00000000000000000000000000000000 P 31cae0b535f04bca833ae9ab7318f4b2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:20:23.417022 4136 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 31cae0b535f04bca833ae9ab7318f4b2 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:20:23.417164 4136 raft_consensus.cc:686] T 00000000000000000000000000000000 P 31cae0b535f04bca833ae9ab7318f4b2 [term 1 LEADER]: Becoming Leader. State: Replica: 31cae0b535f04bca833ae9ab7318f4b2, State: Running, Role: LEADER
I20260929 03:20:23.417234 4131 sys_catalog.cc:564] T 00000000000000000000000000000000 P 31cae0b535f04bca833ae9ab7318f4b2 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:20:23.417284 4136 consensus_queue.cc:237] T 00000000000000000000000000000000 P 31cae0b535f04bca833ae9ab7318f4b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31cae0b535f04bca833ae9ab7318f4b2" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 36419 } }
I20260929 03:20:23.417647 4136 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31cae0b535f04bca833ae9ab7318f4b2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 31cae0b535f04bca833ae9ab7318f4b2. Latest consensus state: current_term: 1 leader_uuid: "31cae0b535f04bca833ae9ab7318f4b2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31cae0b535f04bca833ae9ab7318f4b2" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 36419 } } }
I20260929 03:20:23.417851 4136 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31cae0b535f04bca833ae9ab7318f4b2 [sys.catalog]: This master's current role is: LEADER
I20260929 03:20:23.417781 4137 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31cae0b535f04bca833ae9ab7318f4b2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "31cae0b535f04bca833ae9ab7318f4b2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31cae0b535f04bca833ae9ab7318f4b2" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 36419 } } }
I20260929 03:20:23.418169 4137 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31cae0b535f04bca833ae9ab7318f4b2 [sys.catalog]: This master's current role is: LEADER
W20260929 03:20:23.418967 4152 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 31cae0b535f04bca833ae9ab7318f4b2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 03:20:23.419021 4152 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 03:20:23.419075 4145 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:20:23.419292 4145 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 29 03:20:23 dist-test-slave-c9qh krb5kdc[4050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652023, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.178.126@KRBTEST.COM
I20260929 03:20:23.420921 4145 catalog_manager.cc:1347] Generated new cluster ID: 590b9fa8fbf44c89a1fd76061dafecc8
I20260929 03:20:23.420981 4145 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:20:23.424263 4134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:20:23.412571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40310 (local address 127.20.178.126:36419)
0929 03:20:23.412742 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0929 03:20:23.412751 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 03:20:23.412786 (+ 35us) server_negotiation.cc:407] Connection header received
0929 03:20:23.413314 (+ 528us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:20:23.413322 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:20:23.413378 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:20:23.413538 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:20:23.414114 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:20:23.415562 (+ 1448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:20:23.416484 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:20:23.417134 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:20:23.417554 (+ 420us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:20:23.420766 (+ 3212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:20:23.420770 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:20:23.420770 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:20:23.420782 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:20:23.422727 (+ 1945us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:20:23.423091 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:20:23.423093 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:20:23.423094 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:20:23.423197 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:20:23.423376 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:20:23.423376 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:20:23.423377 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:20:23.423673 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:20:23.423817 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0929 03:20:23.423927 (+ 110us) server_negotiation.cc:299] Negotiation successful
0929 03:20:23.424057 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":96,"threads_started":1}
I20260929 03:20:23.437191 4145 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:20:23.437688 4145 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:20:23.443511 4145 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 31cae0b535f04bca833ae9ab7318f4b2: Generated new TSK 0
I20260929 03:20:23.443717 4145 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.178.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.178.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.178.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.65.keytab.
Entry for principal kudu/127.20.178.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.65.keytab.
I20260929 03:20:23.473580 21193 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuoaHbX/build/release/bin/kudu
/tmp/dist-test-taskuoaHbX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.178.65:0
--local_ip_for_outbound_sockets=127.20.178.65
--webserver_interface=127.20.178.65
--webserver_port=0
--tserver_master_addrs=127.20.178.126:36419
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.65.keytab
--principal=kudu/127.20.178.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.178.84:42735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:20:23.553038 4157 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:20:23.553232 4157 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:20:23.553256 4157 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:20:23.554378 4157 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:20:23.554418 4157 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:20:23.554443 4157 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.178.65
I20260929 03:20:23.555882 4157 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.178.84:42735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.65.keytab
--principal=kudu/127.20.178.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.178.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.178.65
--webserver_port=0
--tserver_master_addrs=127.20.178.126:36419
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4157
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.178.65
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:20:23.556159 4157 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:20:23.556384 4157 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:20:23.561110 4165 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:20:23.561168 4162 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:20:23.561172 4163 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:20:23.561302 4157 server_base.cc:1034] running on GCE node
I20260929 03:20:23.561515 4157 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:20:23.561697 4157 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:20:23.562831 4157 hybrid_clock.cc:648] HybridClock initialized: now 1790652023562819 us; error 28 us; skew 500 ppm
Sep 29 03:20:23 dist-test-slave-c9qh krb5kdc[4050](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652023, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:20:23.565254 4157 init.cc:377] Logged in from keytab as kudu/127.20.178.65@KRBTEST.COM (short username kudu)
I20260929 03:20:23.565667 4157 webserver.cc:460] Webserver started at http://127.20.178.65:33855/ using document root <none> and password file <none>
I20260929 03:20:23.565887 4157 fs_manager.cc:359] Metadata directory not provided
I20260929 03:20:23.565949 4157 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:20:23.566090 4157 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:20:23.567083 4157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/minicluster-data/ts-0/data/instance:
uuid: "f3c3608dbc5146a5affe080bc3253460"
format_stamp: "Formatted at 2026-09-29 03:20:23 on dist-test-slave-c9qh"
server_key: "befe8cef71909dd162379b1998d3d44c"
server_key_iv: "611115b899673601adc691edbc0b0d40"
server_key_version: "encryptionkey@0"
I20260929 03:20:23.567430 4157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/minicluster-data/ts-0/wal/instance:
uuid: "f3c3608dbc5146a5affe080bc3253460"
format_stamp: "Formatted at 2026-09-29 03:20:23 on dist-test-slave-c9qh"
server_key: "befe8cef71909dd162379b1998d3d44c"
server_key_iv: "611115b899673601adc691edbc0b0d40"
server_key_version: "encryptionkey@0"
I20260929 03:20:23.568776 4157 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260929 03:20:23.569458 4172 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:20:23.569615 4157 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 03:20:23.569706 4157 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/minicluster-data/ts-0/data,/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/minicluster-data/ts-0/wal
uuid: "f3c3608dbc5146a5affe080bc3253460"
format_stamp: "Formatted at 2026-09-29 03:20:23 on dist-test-slave-c9qh"
server_key: "befe8cef71909dd162379b1998d3d44c"
server_key_iv: "611115b899673601adc691edbc0b0d40"
server_key_version: "encryptionkey@0"
I20260929 03:20:23.569782 4157 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:20:23.590909 4157 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:20:23.592671 4157 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:20:23.592825 4157 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:20:23.593076 4157 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:20:23.593386 4157 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:20:23.593423 4157 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:20:23.593446 4157 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:20:23.593457 4157 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:20:23.599195 4157 rpc_server.cc:307] RPC server started. Bound to: 127.20.178.65:45779
I20260929 03:20:23.599263 4285 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.178.65:45779 every 8 connection(s)
I20260929 03:20:23.599560 4157 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/minicluster-data/ts-0/data/info.pb
Sep 29 03:20:23 dist-test-slave-c9qh krb5kdc[4050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652023, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.65@KRBTEST.COM for kudu/127.20.178.126@KRBTEST.COM
I20260929 03:20:23.607992 4134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:20:23.600202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.65:46499 (local address 127.20.178.126:36419)
0929 03:20:23.600242 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0929 03:20:23.600245 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:20:23.600456 (+ 211us) server_negotiation.cc:407] Connection header received
0929 03:20:23.600946 (+ 490us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:20:23.600947 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:20:23.600957 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:20:23.601015 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:20:23.601907 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:20:23.602395 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:20:23.603426 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:20:23.603633 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:20:23.603700 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:20:23.605710 (+ 2010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:20:23.605712 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:20:23.605712 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:20:23.605718 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:20:23.606885 (+ 1167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:20:23.607255 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:20:23.607255 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:20:23.607256 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:20:23.607305 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:20:23.607500 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:20:23.607501 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:20:23.607501 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:20:23.607596 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:20:23.607651 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0929 03:20:23.607859 (+ 208us) server_negotiation.cc:299] Negotiation successful
0929 03:20:23.607929 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260929 03:20:23.608178 4288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:20:23.600186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:36419 (local address 127.20.178.65:46499)
0929 03:20:23.600347 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0929 03:20:23.600387 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 03:20:23.600864 (+ 477us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:20:23.601066 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:20:23.601069 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:20:23.601617 (+ 548us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:20:23.601832 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:20:23.601837 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:20:23.602471 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:20:23.602471 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:20:23.603249 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:20:23.603251 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:20:23.603714 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:20:23.603715 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:20:23.603812 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:20:23.604284 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:20:23.604292 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:20:23.605622 (+ 1330us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:20:23.606963 (+ 1341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:20:23.606969 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:20:23.606970 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:20:23.607193 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:20:23.607360 (+ 167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:20:23.607361 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:20:23.607361 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:20:23.607451 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:20:23.607650 (+ 199us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:20:23.607653 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:20:23.607757 (+ 104us) client_negotiation.cc:769] Sending connection context
0929 03:20:23.607835 (+ 78us) client_negotiation.cc:240] Negotiation successful
0929 03:20:23.607946 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":90,"threads_started":1}
I20260929 03:20:23.608438 21193 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuoaHbX/build/release/bin/kudu as pid 4157
I20260929 03:20:23.608541 21193 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/minicluster-data/ts-0/wal/instance
I20260929 03:20:23.608798 21193 external_mini_cluster.cc:1428] Setting key 94d4a6c55bbab7fb481db133b2f9fe66
I20260929 03:20:23.608789 4286 heartbeater.cc:347] Connected to a master server at 127.20.178.126:36419
I20260929 03:20:23.608897 4286 heartbeater.cc:464] Registering TS with master...
I20260929 03:20:23.609100 4286 heartbeater.cc:511] Master 127.20.178.126:36419 requested a full tablet report, sending...
I20260929 03:20:23.609715 4095 ts_manager.cc:194] Registered new tserver with Master: f3c3608dbc5146a5affe080bc3253460 (127.20.178.65:45779)
I20260929 03:20:23.610337 4095 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.178.65@KRBTEST.COM'} at 127.20.178.65:46499
WARNING: no policy specified for kudu/127.20.178.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.178.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.178.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.66.keytab.
Entry for principal kudu/127.20.178.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.66.keytab.
I20260929 03:20:23.631544 21193 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuoaHbX/build/release/bin/kudu
/tmp/dist-test-taskuoaHbX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.178.66:0
--local_ip_for_outbound_sockets=127.20.178.66
--webserver_interface=127.20.178.66
--webserver_port=0
--tserver_master_addrs=127.20.178.126:36419
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.66.keytab
--principal=kudu/127.20.178.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.178.84:42735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:20:23.710605 4291 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:20:23.710834 4291 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:20:23.710871 4291 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:20:23.712064 4291 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:20:23.712137 4291 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:20:23.712186 4291 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.178.66
I20260929 03:20:23.713672 4291 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.178.84:42735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.66.keytab
--principal=kudu/127.20.178.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.178.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.178.66
--webserver_port=0
--tserver_master_addrs=127.20.178.126:36419
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4291
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.178.66
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:20:23.713933 4291 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:20:23.714154 4291 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:20:23.718989 4296 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:20:23.718997 4297 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:20:23.719206 4291 server_base.cc:1034] running on GCE node
W20260929 03:20:23.718990 4299 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:20:23.719450 4291 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:20:23.719678 4291 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:20:23.720836 4291 hybrid_clock.cc:648] HybridClock initialized: now 1790652023720820 us; error 30 us; skew 500 ppm
Sep 29 03:20:23 dist-test-slave-c9qh krb5kdc[4050](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652023, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:20:23.723242 4291 init.cc:377] Logged in from keytab as kudu/127.20.178.66@KRBTEST.COM (short username kudu)
I20260929 03:20:23.723606 4291 webserver.cc:460] Webserver started at http://127.20.178.66:34237/ using document root <none> and password file <none>
I20260929 03:20:23.723791 4291 fs_manager.cc:359] Metadata directory not provided
I20260929 03:20:23.723830 4291 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:20:23.723914 4291 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:20:23.724944 4291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/minicluster-data/ts-1/data/instance:
uuid: "9978d1b39862456881b65ee097849482"
format_stamp: "Formatted at 2026-09-29 03:20:23 on dist-test-slave-c9qh"
server_key: "aaefa21c358dc74638c24d23eda049bd"
server_key_iv: "82497572b79b0a0a6d8302733bc33410"
server_key_version: "encryptionkey@0"
I20260929 03:20:23.725309 4291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/minicluster-data/ts-1/wal/instance:
uuid: "9978d1b39862456881b65ee097849482"
format_stamp: "Formatted at 2026-09-29 03:20:23 on dist-test-slave-c9qh"
server_key: "aaefa21c358dc74638c24d23eda049bd"
server_key_iv: "82497572b79b0a0a6d8302733bc33410"
server_key_version: "encryptionkey@0"
I20260929 03:20:23.726502 4291 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 03:20:23.727118 4306 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:20:23.727283 4291 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 03:20:23.727365 4291 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/minicluster-data/ts-1/data,/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/minicluster-data/ts-1/wal
uuid: "9978d1b39862456881b65ee097849482"
format_stamp: "Formatted at 2026-09-29 03:20:23 on dist-test-slave-c9qh"
server_key: "aaefa21c358dc74638c24d23eda049bd"
server_key_iv: "82497572b79b0a0a6d8302733bc33410"
server_key_version: "encryptionkey@0"
I20260929 03:20:23.727430 4291 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:20:23.745517 4291 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:20:23.747712 4291 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:20:23.747864 4291 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:20:23.748070 4291 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:20:23.748368 4291 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:20:23.748417 4291 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:20:23.748451 4291 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:20:23.748477 4291 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:20:23.753924 4291 rpc_server.cc:307] RPC server started. Bound to: 127.20.178.66:43451
I20260929 03:20:23.753993 4419 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.178.66:43451 every 8 connection(s)
I20260929 03:20:23.754299 4291 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/minicluster-data/ts-1/data/info.pb
I20260929 03:20:23.755630 21193 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuoaHbX/build/release/bin/kudu as pid 4291
I20260929 03:20:23.755731 21193 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/minicluster-data/ts-1/wal/instance
I20260929 03:20:23.755934 21193 external_mini_cluster.cc:1428] Setting key 80c588361fa7ed6c12e86709c78a6397
Sep 29 03:20:23 dist-test-slave-c9qh krb5kdc[4050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652023, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.66@KRBTEST.COM for kudu/127.20.178.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.20.178.67@KRBTEST.COM; defaulting to no policy
I20260929 03:20:23.764194 4134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:20:23.754987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.66:49359 (local address 127.20.178.126:36419)
0929 03:20:23.755049 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0929 03:20:23.755051 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:20:23.755221 (+ 170us) server_negotiation.cc:407] Connection header received
0929 03:20:23.755847 (+ 626us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:20:23.755848 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:20:23.755858 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:20:23.755909 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:20:23.756974 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:20:23.757465 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:20:23.758682 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:20:23.758993 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:20:23.759058 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:20:23.761445 (+ 2387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:20:23.761446 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:20:23.761447 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:20:23.761455 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:20:23.762797 (+ 1342us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:20:23.763215 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:20:23.763216 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:20:23.763217 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:20:23.763273 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:20:23.763518 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:20:23.763519 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:20:23.763519 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:20:23.763628 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:20:23.763715 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0929 03:20:23.764029 (+ 314us) server_negotiation.cc:299] Negotiation successful
0929 03:20:23.764105 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260929 03:20:23.764375 4422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:20:23.754952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:36419 (local address 127.20.178.66:49359)
0929 03:20:23.755103 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0929 03:20:23.755143 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 03:20:23.755746 (+ 603us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:20:23.755980 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:20:23.755985 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:20:23.756589 (+ 604us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:20:23.756890 (+ 301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:20:23.756897 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:20:23.757573 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:20:23.757574 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:20:23.758576 (+ 1002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:20:23.758579 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:20:23.759086 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:20:23.759087 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:20:23.759243 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:20:23.759725 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:20:23.759735 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:20:23.761350 (+ 1615us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:20:23.762895 (+ 1545us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:20:23.762903 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:20:23.762905 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:20:23.763123 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:20:23.763342 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:20:23.763343 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:20:23.763343 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:20:23.763449 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:20:23.763710 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:20:23.763717 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:20:23.763887 (+ 170us) client_negotiation.cc:769] Sending connection context
0929 03:20:23.763993 (+ 106us) client_negotiation.cc:240] Negotiation successful
0929 03:20:23.764141 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":80,"threads_started":1}
I20260929 03:20:23.764920 4420 heartbeater.cc:347] Connected to a master server at 127.20.178.126:36419
I20260929 03:20:23.765029 4420 heartbeater.cc:464] Registering TS with master...
I20260929 03:20:23.765199 4420 heartbeater.cc:511] Master 127.20.178.126:36419 requested a full tablet report, sending...
I20260929 03:20:23.765563 4095 ts_manager.cc:194] Registered new tserver with Master: 9978d1b39862456881b65ee097849482 (127.20.178.66:43451)
I20260929 03:20:23.765946 4095 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.178.66@KRBTEST.COM'} at 127.20.178.66:49359
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.178.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.178.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.67.keytab.
Entry for principal kudu/127.20.178.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.67.keytab.
I20260929 03:20:23.779198 21193 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuoaHbX/build/release/bin/kudu
/tmp/dist-test-taskuoaHbX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.178.67:0
--local_ip_for_outbound_sockets=127.20.178.67
--webserver_interface=127.20.178.67
--webserver_port=0
--tserver_master_addrs=127.20.178.126:36419
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.67.keytab
--principal=kudu/127.20.178.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.178.84:42735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:20:23.859735 4425 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:20:23.859923 4425 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:20:23.859946 4425 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:20:23.861097 4425 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:20:23.861138 4425 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:20:23.861194 4425 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.178.67
I20260929 03:20:23.862584 4425 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.178.84:42735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.67.keytab
--principal=kudu/127.20.178.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.178.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.178.67
--webserver_port=0
--tserver_master_addrs=127.20.178.126:36419
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4425
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.178.67
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:20:23.862826 4425 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:20:23.863042 4425 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:20:23.867807 4430 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:20:23.867945 4425 server_base.cc:1034] running on GCE node
W20260929 03:20:23.867808 4433 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:20:23.867874 4431 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:20:23.868263 4425 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:20:23.868458 4425 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:20:23.869630 4425 hybrid_clock.cc:648] HybridClock initialized: now 1790652023869593 us; error 50 us; skew 500 ppm
Sep 29 03:20:23 dist-test-slave-c9qh krb5kdc[4050](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652023, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:20:23.871978 4425 init.cc:377] Logged in from keytab as kudu/127.20.178.67@KRBTEST.COM (short username kudu)
I20260929 03:20:23.872361 4425 webserver.cc:460] Webserver started at http://127.20.178.67:45173/ using document root <none> and password file <none>
I20260929 03:20:23.872591 4425 fs_manager.cc:359] Metadata directory not provided
I20260929 03:20:23.872673 4425 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:20:23.872781 4425 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:20:23.873778 4425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/minicluster-data/ts-2/data/instance:
uuid: "15b1f41f721649dd8a97d314979ac521"
format_stamp: "Formatted at 2026-09-29 03:20:23 on dist-test-slave-c9qh"
server_key: "dca81f8dcb62cb70aa55e2579b502690"
server_key_iv: "3565d2f4dcd71b8419b834ddb84a7a6d"
server_key_version: "encryptionkey@0"
I20260929 03:20:23.874071 4425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/minicluster-data/ts-2/wal/instance:
uuid: "15b1f41f721649dd8a97d314979ac521"
format_stamp: "Formatted at 2026-09-29 03:20:23 on dist-test-slave-c9qh"
server_key: "dca81f8dcb62cb70aa55e2579b502690"
server_key_iv: "3565d2f4dcd71b8419b834ddb84a7a6d"
server_key_version: "encryptionkey@0"
I20260929 03:20:23.875272 4425 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 03:20:23.875936 4440 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:20:23.876138 4425 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 03:20:23.876242 4425 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/minicluster-data/ts-2/data,/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/minicluster-data/ts-2/wal
uuid: "15b1f41f721649dd8a97d314979ac521"
format_stamp: "Formatted at 2026-09-29 03:20:23 on dist-test-slave-c9qh"
server_key: "dca81f8dcb62cb70aa55e2579b502690"
server_key_iv: "3565d2f4dcd71b8419b834ddb84a7a6d"
server_key_version: "encryptionkey@0"
I20260929 03:20:23.876312 4425 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:20:23.889120 4425 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:20:23.891332 4425 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:20:23.891484 4425 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:20:23.891687 4425 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:20:23.891984 4425 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:20:23.892036 4425 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:20:23.892077 4425 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:20:23.892105 4425 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:20:23.898090 4425 rpc_server.cc:307] RPC server started. Bound to: 127.20.178.67:33143
I20260929 03:20:23.898161 4553 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.178.67:33143 every 8 connection(s)
I20260929 03:20:23.898494 4425 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/minicluster-data/ts-2/data/info.pb
I20260929 03:20:23.903451 21193 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuoaHbX/build/release/bin/kudu as pid 4425
I20260929 03:20:23.903551 21193 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/minicluster-data/ts-2/wal/instance
I20260929 03:20:23.903734 21193 external_mini_cluster.cc:1428] Setting key f68235a7e148e15a807fc87db17a0cba
Sep 29 03:20:23 dist-test-slave-c9qh krb5kdc[4050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652023, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.67@KRBTEST.COM for kudu/127.20.178.126@KRBTEST.COM
I20260929 03:20:23.907433 4134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:20:23.899177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.67:58597 (local address 127.20.178.126:36419)
0929 03:20:23.899236 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0929 03:20:23.899238 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:20:23.899439 (+ 201us) server_negotiation.cc:407] Connection header received
0929 03:20:23.899992 (+ 553us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:20:23.899994 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:20:23.900007 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:20:23.900054 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:20:23.901132 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:20:23.901661 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:20:23.902626 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:20:23.902837 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:20:23.902909 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:20:23.905233 (+ 2324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:20:23.905234 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:20:23.905235 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:20:23.905240 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:20:23.906340 (+ 1100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:20:23.906680 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:20:23.906681 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:20:23.906681 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:20:23.906731 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:20:23.906946 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:20:23.906946 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:20:23.906947 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:20:23.907060 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:20:23.907108 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0929 03:20:23.907320 (+ 212us) server_negotiation.cc:299] Negotiation successful
0929 03:20:23.907374 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260929 03:20:23.907600 4556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:20:23.899160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:36419 (local address 127.20.178.67:58597)
0929 03:20:23.899331 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0929 03:20:23.899364 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0929 03:20:23.899879 (+ 515us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:20:23.900139 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:20:23.900143 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:20:23.900788 (+ 645us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:20:23.901035 (+ 247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:20:23.901041 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:20:23.901752 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:20:23.901753 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:20:23.902537 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:20:23.902539 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:20:23.902928 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:20:23.902929 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:20:23.903053 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:20:23.903661 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:20:23.903669 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:20:23.905170 (+ 1501us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:20:23.906404 (+ 1234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:20:23.906410 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:20:23.906411 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:20:23.906617 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:20:23.906789 (+ 172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:20:23.906790 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:20:23.906790 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:20:23.906897 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:20:23.907110 (+ 213us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:20:23.907113 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:20:23.907235 (+ 122us) client_negotiation.cc:769] Sending connection context
0929 03:20:23.907311 (+ 76us) client_negotiation.cc:240] Negotiation successful
0929 03:20:23.907417 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":85,"threads_started":1}
I20260929 03:20:23.908030 4554 heartbeater.cc:347] Connected to a master server at 127.20.178.126:36419
I20260929 03:20:23.908123 4554 heartbeater.cc:464] Registering TS with master...
I20260929 03:20:23.908254 4554 heartbeater.cc:511] Master 127.20.178.126:36419 requested a full tablet report, sending...
I20260929 03:20:23.908596 4095 ts_manager.cc:194] Registered new tserver with Master: 15b1f41f721649dd8a97d314979ac521 (127.20.178.67:33143)
I20260929 03:20:23.909022 4095 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.178.67@KRBTEST.COM'} at 127.20.178.67:58597
I20260929 03:20:23.915366 21193 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 03:20:23.922223 4134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:20:23.916691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40316 (local address 127.20.178.126:36419)
0929 03:20:23.916725 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0929 03:20:23.916727 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:20:23.916895 (+ 168us) server_negotiation.cc:407] Connection header received
0929 03:20:23.916931 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:20:23.916932 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:20:23.916940 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:20:23.916991 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:20:23.917543 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:20:23.918098 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:20:23.918906 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:20:23.919078 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:20:23.919142 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:20:23.920197 (+ 1055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:20:23.920197 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:20:23.920198 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:20:23.920202 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:20:23.921336 (+ 1134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:20:23.921621 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:20:23.921622 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:20:23.921622 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:20:23.921667 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:20:23.921835 (+ 168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:20:23.921835 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:20:23.921835 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:20:23.921951 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:20:23.922023 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0929 03:20:23.922103 (+ 80us) server_negotiation.cc:299] Negotiation successful
0929 03:20:23.922168 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260929 03:20:23.923503 4095 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40316:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 03:20:23.925129 4095 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 03:20:23.930855 4563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:20:23.927017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.66:43451 (local address 127.0.0.1:41912)
0929 03:20:23.927347 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0929 03:20:23.927355 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0929 03:20:23.927390 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:20:23.927630 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:20:23.927633 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:20:23.927635 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:20:23.927689 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:20:23.927690 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:20:23.928610 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:20:23.928611 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:20:23.929724 (+ 1113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:20:23.929726 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:20:23.930565 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:20:23.930566 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:20:23.930657 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:20:23.930671 (+ 14us) client_negotiation.cc:769] Sending connection context
0929 03:20:23.930743 (+ 72us) client_negotiation.cc:240] Negotiation successful
0929 03:20:23.930787 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":45,"threads_started":1}
I20260929 03:20:23.930886 4566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:20:23.927278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41912 (local address 127.20.178.66:43451)
0929 03:20:23.927452 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0929 03:20:23.927457 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:20:23.927467 (+ 10us) server_negotiation.cc:407] Connection header received
0929 03:20:23.927504 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:20:23.927506 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:20:23.927519 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:20:23.927611 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:20:23.927789 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:20:23.928513 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:20:23.929815 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:20:23.930481 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:20:23.930553 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:20:23.930607 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0929 03:20:23.930766 (+ 159us) server_negotiation.cc:299] Negotiation successful
0929 03:20:23.930808 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":99,"threads_started":1}
I20260929 03:20:23.931257 4562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:20:23.927004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.65:45779 (local address 127.0.0.1:48090)
0929 03:20:23.927157 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0929 03:20:23.927201 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 03:20:23.927261 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:20:23.927691 (+ 430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:20:23.927692 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:20:23.927693 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:20:23.927764 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:20:23.927767 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:20:23.929091 (+ 1324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:20:23.929092 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:20:23.930119 (+ 1027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:20:23.930121 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:20:23.931028 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:20:23.931029 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:20:23.931098 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:20:23.931101 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 03:20:23.931148 (+ 47us) client_negotiation.cc:240] Negotiation successful
0929 03:20:23.931192 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"mutex_wait_us":4,"thread_start_us":45,"threads_started":1}
I20260929 03:20:23.931531 4565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:20:23.927202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48090 (local address 127.20.178.65:45779)
0929 03:20:23.927506 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0929 03:20:23.927511 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:20:23.927521 (+ 10us) server_negotiation.cc:407] Connection header received
0929 03:20:23.927560 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:20:23.927562 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:20:23.927576 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:20:23.927844 (+ 268us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:20:23.927910 (+ 66us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:20:23.928992 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:20:23.930199 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:20:23.930954 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:20:23.931026 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:20:23.931078 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0929 03:20:23.931419 (+ 341us) server_negotiation.cc:299] Negotiation successful
0929 03:20:23.931458 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267,"thread_start_us":76,"threads_started":1}
I20260929 03:20:23.931692 4354 tablet_service.cc:1467] Processing CreateTablet for tablet ebd9c98e971a40b3931625b26ff4b3b8 (DEFAULT_TABLE table=test-table [id=0ac7856ecacb454a87cad08ba1eb5d91]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:20:23.932054 4354 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ebd9c98e971a40b3931625b26ff4b3b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:20:23.932317 4564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:20:23.927004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.67:33143 (local address 127.0.0.1:36124)
0929 03:20:23.927249 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0929 03:20:23.927260 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 03:20:23.927309 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:20:23.927682 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:20:23.927683 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:20:23.927685 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:20:23.927764 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:20:23.927766 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:20:23.930188 (+ 2422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:20:23.930188 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:20:23.931051 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:20:23.931053 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:20:23.932112 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:20:23.932113 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:20:23.932179 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:20:23.932184 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 03:20:23.932216 (+ 32us) client_negotiation.cc:240] Negotiation successful
0929 03:20:23.932252 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"mutex_wait_us":5,"thread_start_us":45,"threads_started":1}
I20260929 03:20:23.932227 4220 tablet_service.cc:1467] Processing CreateTablet for tablet ebd9c98e971a40b3931625b26ff4b3b8 (DEFAULT_TABLE table=test-table [id=0ac7856ecacb454a87cad08ba1eb5d91]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:20:23.932549 4220 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ebd9c98e971a40b3931625b26ff4b3b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:20:23.932709 4567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:20:23.927365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36124 (local address 127.20.178.67:33143)
0929 03:20:23.927521 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0929 03:20:23.927525 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:20:23.927533 (+ 8us) server_negotiation.cc:407] Connection header received
0929 03:20:23.927561 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:20:23.927563 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:20:23.927576 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:20:23.927844 (+ 268us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:20:23.927904 (+ 60us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:20:23.928983 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:20:23.931146 (+ 2163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:20:23.932011 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:20:23.932482 (+ 471us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:20:23.932541 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0929 03:20:23.932566 (+ 25us) server_negotiation.cc:299] Negotiation successful
0929 03:20:23.932606 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":91,"threads_started":1}
I20260929 03:20:23.933319 4488 tablet_service.cc:1467] Processing CreateTablet for tablet ebd9c98e971a40b3931625b26ff4b3b8 (DEFAULT_TABLE table=test-table [id=0ac7856ecacb454a87cad08ba1eb5d91]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:20:23.933630 4488 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ebd9c98e971a40b3931625b26ff4b3b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:20:23.934881 4568 tablet_bootstrap.cc:492] T ebd9c98e971a40b3931625b26ff4b3b8 P 9978d1b39862456881b65ee097849482: Bootstrap starting.
I20260929 03:20:23.935470 4568 tablet_bootstrap.cc:654] T ebd9c98e971a40b3931625b26ff4b3b8 P 9978d1b39862456881b65ee097849482: Neither blocks nor log segments found. Creating new log.
I20260929 03:20:23.935890 4569 tablet_bootstrap.cc:492] T ebd9c98e971a40b3931625b26ff4b3b8 P f3c3608dbc5146a5affe080bc3253460: Bootstrap starting.
I20260929 03:20:23.936050 4568 log.cc:824] T ebd9c98e971a40b3931625b26ff4b3b8 P 9978d1b39862456881b65ee097849482: Log is configured to *not* fsync() on all Append() calls
I20260929 03:20:23.936568 4569 tablet_bootstrap.cc:654] T ebd9c98e971a40b3931625b26ff4b3b8 P f3c3608dbc5146a5affe080bc3253460: Neither blocks nor log segments found. Creating new log.
I20260929 03:20:23.936812 4571 tablet_bootstrap.cc:492] T ebd9c98e971a40b3931625b26ff4b3b8 P 15b1f41f721649dd8a97d314979ac521: Bootstrap starting.
I20260929 03:20:23.937204 4568 tablet_bootstrap.cc:492] T ebd9c98e971a40b3931625b26ff4b3b8 P 9978d1b39862456881b65ee097849482: No bootstrap required, opened a new log
I20260929 03:20:23.937196 4569 log.cc:824] T ebd9c98e971a40b3931625b26ff4b3b8 P f3c3608dbc5146a5affe080bc3253460: Log is configured to *not* fsync() on all Append() calls
I20260929 03:20:23.937271 4568 ts_tablet_manager.cc:1397] T ebd9c98e971a40b3931625b26ff4b3b8 P 9978d1b39862456881b65ee097849482: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 03:20:23.937506 4571 tablet_bootstrap.cc:654] T ebd9c98e971a40b3931625b26ff4b3b8 P 15b1f41f721649dd8a97d314979ac521: Neither blocks nor log segments found. Creating new log.
I20260929 03:20:23.937970 4571 log.cc:824] T ebd9c98e971a40b3931625b26ff4b3b8 P 15b1f41f721649dd8a97d314979ac521: Log is configured to *not* fsync() on all Append() calls
I20260929 03:20:23.937978 4569 tablet_bootstrap.cc:492] T ebd9c98e971a40b3931625b26ff4b3b8 P f3c3608dbc5146a5affe080bc3253460: No bootstrap required, opened a new log
I20260929 03:20:23.938058 4569 ts_tablet_manager.cc:1397] T ebd9c98e971a40b3931625b26ff4b3b8 P f3c3608dbc5146a5affe080bc3253460: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 03:20:23.938616 4571 tablet_bootstrap.cc:492] T ebd9c98e971a40b3931625b26ff4b3b8 P 15b1f41f721649dd8a97d314979ac521: No bootstrap required, opened a new log
I20260929 03:20:23.938700 4571 ts_tablet_manager.cc:1397] T ebd9c98e971a40b3931625b26ff4b3b8 P 15b1f41f721649dd8a97d314979ac521: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 03:20:23.939203 4568 raft_consensus.cc:348] T ebd9c98e971a40b3931625b26ff4b3b8 P 9978d1b39862456881b65ee097849482 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c3608dbc5146a5affe080bc3253460" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 45779 } } peers { permanent_uuid: "15b1f41f721649dd8a97d314979ac521" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 33143 } } peers { permanent_uuid: "9978d1b39862456881b65ee097849482" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 43451 } }
I20260929 03:20:23.939381 4568 raft_consensus.cc:374] T ebd9c98e971a40b3931625b26ff4b3b8 P 9978d1b39862456881b65ee097849482 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:20:23.939419 4568 raft_consensus.cc:729] T ebd9c98e971a40b3931625b26ff4b3b8 P 9978d1b39862456881b65ee097849482 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9978d1b39862456881b65ee097849482, State: Initialized, Role: FOLLOWER
I20260929 03:20:23.939548 4568 consensus_queue.cc:260] T ebd9c98e971a40b3931625b26ff4b3b8 P 9978d1b39862456881b65ee097849482 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c3608dbc5146a5affe080bc3253460" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 45779 } } peers { permanent_uuid: "15b1f41f721649dd8a97d314979ac521" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 33143 } } peers { permanent_uuid: "9978d1b39862456881b65ee097849482" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 43451 } }
I20260929 03:20:23.939877 4568 ts_tablet_manager.cc:1428] T ebd9c98e971a40b3931625b26ff4b3b8 P 9978d1b39862456881b65ee097849482: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260929 03:20:23.940008 4420 heartbeater.cc:503] Master 127.20.178.126:36419 was elected leader, sending a full tablet report...
I20260929 03:20:23.940016 4569 raft_consensus.cc:348] T ebd9c98e971a40b3931625b26ff4b3b8 P f3c3608dbc5146a5affe080bc3253460 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c3608dbc5146a5affe080bc3253460" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 45779 } } peers { permanent_uuid: "15b1f41f721649dd8a97d314979ac521" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 33143 } } peers { permanent_uuid: "9978d1b39862456881b65ee097849482" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 43451 } }
I20260929 03:20:23.940202 4569 raft_consensus.cc:374] T ebd9c98e971a40b3931625b26ff4b3b8 P f3c3608dbc5146a5affe080bc3253460 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:20:23.940244 4569 raft_consensus.cc:729] T ebd9c98e971a40b3931625b26ff4b3b8 P f3c3608dbc5146a5affe080bc3253460 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3c3608dbc5146a5affe080bc3253460, State: Initialized, Role: FOLLOWER
I20260929 03:20:23.940362 4569 consensus_queue.cc:260] T ebd9c98e971a40b3931625b26ff4b3b8 P f3c3608dbc5146a5affe080bc3253460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c3608dbc5146a5affe080bc3253460" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 45779 } } peers { permanent_uuid: "15b1f41f721649dd8a97d314979ac521" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 33143 } } peers { permanent_uuid: "9978d1b39862456881b65ee097849482" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 43451 } }
I20260929 03:20:23.940280 4571 raft_consensus.cc:348] T ebd9c98e971a40b3931625b26ff4b3b8 P 15b1f41f721649dd8a97d314979ac521 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c3608dbc5146a5affe080bc3253460" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 45779 } } peers { permanent_uuid: "15b1f41f721649dd8a97d314979ac521" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 33143 } } peers { permanent_uuid: "9978d1b39862456881b65ee097849482" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 43451 } }
I20260929 03:20:23.940462 4571 raft_consensus.cc:374] T ebd9c98e971a40b3931625b26ff4b3b8 P 15b1f41f721649dd8a97d314979ac521 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:20:23.940502 4571 raft_consensus.cc:729] T ebd9c98e971a40b3931625b26ff4b3b8 P 15b1f41f721649dd8a97d314979ac521 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15b1f41f721649dd8a97d314979ac521, State: Initialized, Role: FOLLOWER
I20260929 03:20:23.940629 4569 ts_tablet_manager.cc:1428] T ebd9c98e971a40b3931625b26ff4b3b8 P f3c3608dbc5146a5affe080bc3253460: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260929 03:20:23.940685 4571 consensus_queue.cc:260] T ebd9c98e971a40b3931625b26ff4b3b8 P 15b1f41f721649dd8a97d314979ac521 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c3608dbc5146a5affe080bc3253460" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 45779 } } peers { permanent_uuid: "15b1f41f721649dd8a97d314979ac521" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 33143 } } peers { permanent_uuid: "9978d1b39862456881b65ee097849482" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 43451 } }
I20260929 03:20:23.940778 4286 heartbeater.cc:503] Master 127.20.178.126:36419 was elected leader, sending a full tablet report...
I20260929 03:20:23.940981 4571 ts_tablet_manager.cc:1428] T ebd9c98e971a40b3931625b26ff4b3b8 P 15b1f41f721649dd8a97d314979ac521: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 03:20:23.941216 4554 heartbeater.cc:503] Master 127.20.178.126:36419 was elected leader, sending a full tablet report...
I20260929 03:20:23.975811 4574 raft_consensus.cc:482] T ebd9c98e971a40b3931625b26ff4b3b8 P 9978d1b39862456881b65ee097849482 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:20:23.975951 4574 raft_consensus.cc:504] T ebd9c98e971a40b3931625b26ff4b3b8 P 9978d1b39862456881b65ee097849482 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c3608dbc5146a5affe080bc3253460" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 45779 } } peers { permanent_uuid: "15b1f41f721649dd8a97d314979ac521" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 33143 } } peers { permanent_uuid: "9978d1b39862456881b65ee097849482" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 43451 } }
I20260929 03:20:23.976325 4574 leader_election.cc:288] T ebd9c98e971a40b3931625b26ff4b3b8 P 9978d1b39862456881b65ee097849482 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f3c3608dbc5146a5affe080bc3253460 (127.20.178.65:45779), 15b1f41f721649dd8a97d314979ac521 (127.20.178.67:33143)
I20260929 03:20:23.980208 4567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:20:23.976553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.66:40939 (local address 127.20.178.67:33143)
0929 03:20:23.976597 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0929 03:20:23.976599 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:20:23.976778 (+ 179us) server_negotiation.cc:407] Connection header received
0929 03:20:23.976828 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:20:23.976829 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:20:23.976839 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:20:23.976892 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:20:23.977030 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:20:23.977729 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:20:23.979263 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:20:23.979845 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:20:23.979913 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:20:23.979921 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0929 03:20:23.980091 (+ 170us) server_negotiation.cc:299] Negotiation successful
0929 03:20:23.980143 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260929 03:20:23.980244 4565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:20:23.976623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.66:43799 (local address 127.20.178.65:45779)
0929 03:20:23.976678 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0929 03:20:23.976680 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:20:23.976689 (+ 9us) server_negotiation.cc:407] Connection header received
0929 03:20:23.976717 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:20:23.976718 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:20:23.976727 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:20:23.976803 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:20:23.976979 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:20:23.977748 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:20:23.979256 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:20:23.979853 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:20:23.979913 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:20:23.979921 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0929 03:20:23.980144 (+ 223us) server_negotiation.cc:299] Negotiation successful
0929 03:20:23.980179 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260929 03:20:23.980240 4422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:20:23.976487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.65:45779 (local address 127.20.178.66:43799)
0929 03:20:23.976530 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0929 03:20:23.976542 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 03:20:23.976589 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:20:23.976816 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:20:23.976817 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:20:23.976819 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:20:23.976907 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:20:23.976909 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:20:23.977816 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:20:23.977816 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:20:23.979175 (+ 1359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:20:23.979176 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:20:23.979934 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:20:23.979935 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:20:23.980019 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:20:23.980021 (+ 2us) client_negotiation.cc:769] Sending connection context
0929 03:20:23.980077 (+ 56us) client_negotiation.cc:240] Negotiation successful
0929 03:20:23.980122 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21,"mutex_wait_us":67}
I20260929 03:20:23.980249 4577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:20:23.976489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.67:33143 (local address 127.20.178.66:40939)
0929 03:20:23.976717 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0929 03:20:23.976724 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0929 03:20:23.976764 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:20:23.976902 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:20:23.976902 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:20:23.976904 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:20:23.976965 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:20:23.976967 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:20:23.977816 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:20:23.977816 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:20:23.979165 (+ 1349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:20:23.979169 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:20:23.979934 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:20:23.979935 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:20:23.980016 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:20:23.980020 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 03:20:23.980077 (+ 57us) client_negotiation.cc:240] Negotiation successful
0929 03:20:23.980122 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216,"mutex_wait_us":98,"thread_start_us":111,"threads_started":1}
I20260929 03:20:23.980685 4240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebd9c98e971a40b3931625b26ff4b3b8" candidate_uuid: "9978d1b39862456881b65ee097849482" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3c3608dbc5146a5affe080bc3253460" is_pre_election: true
I20260929 03:20:23.980845 4240 raft_consensus.cc:2454] T ebd9c98e971a40b3931625b26ff4b3b8 P f3c3608dbc5146a5affe080bc3253460 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9978d1b39862456881b65ee097849482 in term 0.
I20260929 03:20:23.980842 4508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebd9c98e971a40b3931625b26ff4b3b8" candidate_uuid: "9978d1b39862456881b65ee097849482" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15b1f41f721649dd8a97d314979ac521" is_pre_election: true
I20260929 03:20:23.980986 4508 raft_consensus.cc:2454] T ebd9c98e971a40b3931625b26ff4b3b8 P 15b1f41f721649dd8a97d314979ac521 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9978d1b39862456881b65ee097849482 in term 0.
I20260929 03:20:23.981063 4307 leader_election.cc:302] T ebd9c98e971a40b3931625b26ff4b3b8 P 9978d1b39862456881b65ee097849482 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9978d1b39862456881b65ee097849482, f3c3608dbc5146a5affe080bc3253460; no voters:
I20260929 03:20:23.981254 4574 raft_consensus.cc:2781] T ebd9c98e971a40b3931625b26ff4b3b8 P 9978d1b39862456881b65ee097849482 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 03:20:23.981300 4574 raft_consensus.cc:482] T ebd9c98e971a40b3931625b26ff4b3b8 P 9978d1b39862456881b65ee097849482 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 03:20:23.981316 4574 raft_consensus.cc:3037] T ebd9c98e971a40b3931625b26ff4b3b8 P 9978d1b39862456881b65ee097849482 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:20:23.982017 4574 raft_consensus.cc:504] T ebd9c98e971a40b3931625b26ff4b3b8 P 9978d1b39862456881b65ee097849482 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c3608dbc5146a5affe080bc3253460" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 45779 } } peers { permanent_uuid: "15b1f41f721649dd8a97d314979ac521" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 33143 } } peers { permanent_uuid: "9978d1b39862456881b65ee097849482" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 43451 } }
I20260929 03:20:23.982152 4574 leader_election.cc:288] T ebd9c98e971a40b3931625b26ff4b3b8 P 9978d1b39862456881b65ee097849482 [CANDIDATE]: Term 1 election: Requested vote from peers f3c3608dbc5146a5affe080bc3253460 (127.20.178.65:45779), 15b1f41f721649dd8a97d314979ac521 (127.20.178.67:33143)
I20260929 03:20:23.982302 4508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebd9c98e971a40b3931625b26ff4b3b8" candidate_uuid: "9978d1b39862456881b65ee097849482" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15b1f41f721649dd8a97d314979ac521"
I20260929 03:20:23.982292 4240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebd9c98e971a40b3931625b26ff4b3b8" candidate_uuid: "9978d1b39862456881b65ee097849482" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3c3608dbc5146a5affe080bc3253460"
I20260929 03:20:23.982386 4508 raft_consensus.cc:3037] T ebd9c98e971a40b3931625b26ff4b3b8 P 15b1f41f721649dd8a97d314979ac521 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:20:23.982386 4240 raft_consensus.cc:3037] T ebd9c98e971a40b3931625b26ff4b3b8 P f3c3608dbc5146a5affe080bc3253460 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:20:23.983063 4508 raft_consensus.cc:2454] T ebd9c98e971a40b3931625b26ff4b3b8 P 15b1f41f721649dd8a97d314979ac521 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9978d1b39862456881b65ee097849482 in term 1.
I20260929 03:20:23.983134 4240 raft_consensus.cc:2454] T ebd9c98e971a40b3931625b26ff4b3b8 P f3c3608dbc5146a5affe080bc3253460 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9978d1b39862456881b65ee097849482 in term 1.
I20260929 03:20:23.983319 4307 leader_election.cc:302] T ebd9c98e971a40b3931625b26ff4b3b8 P 9978d1b39862456881b65ee097849482 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9978d1b39862456881b65ee097849482, f3c3608dbc5146a5affe080bc3253460; no voters:
I20260929 03:20:23.983423 4574 raft_consensus.cc:2781] T ebd9c98e971a40b3931625b26ff4b3b8 P 9978d1b39862456881b65ee097849482 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:20:23.983557 4574 raft_consensus.cc:686] T ebd9c98e971a40b3931625b26ff4b3b8 P 9978d1b39862456881b65ee097849482 [term 1 LEADER]: Becoming Leader. State: Replica: 9978d1b39862456881b65ee097849482, State: Running, Role: LEADER
I20260929 03:20:23.983645 4574 consensus_queue.cc:237] T ebd9c98e971a40b3931625b26ff4b3b8 P 9978d1b39862456881b65ee097849482 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c3608dbc5146a5affe080bc3253460" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 45779 } } peers { permanent_uuid: "15b1f41f721649dd8a97d314979ac521" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 33143 } } peers { permanent_uuid: "9978d1b39862456881b65ee097849482" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 43451 } }
I20260929 03:20:23.984397 4095 catalog_manager.cc:5495] T ebd9c98e971a40b3931625b26ff4b3b8 P 9978d1b39862456881b65ee097849482 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9978d1b39862456881b65ee097849482 (127.20.178.66). New cstate: current_term: 1 leader_uuid: "9978d1b39862456881b65ee097849482" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c3608dbc5146a5affe080bc3253460" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 45779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "15b1f41f721649dd8a97d314979ac521" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 33143 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9978d1b39862456881b65ee097849482" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 43451 } health_report { overall_health: HEALTHY } } }
I20260929 03:20:23.989638 4566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:20:23.987427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41928 (local address 127.20.178.66:43451)
0929 03:20:23.987485 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0929 03:20:23.987487 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:20:23.987493 (+ 6us) server_negotiation.cc:407] Connection header received
0929 03:20:23.987547 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:20:23.987548 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:20:23.987562 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:20:23.987618 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:20:23.987747 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:20:23.988184 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:20:23.988992 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:20:23.989170 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:20:23.989221 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:20:23.989340 (+ 119us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:20:23.989422 (+ 82us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:20:23.989479 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0929 03:20:23.989510 (+ 31us) server_negotiation.cc:299] Negotiation successful
0929 03:20:23.989525 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260929 03:20:23.991175 4508 raft_consensus.cc:1261] T ebd9c98e971a40b3931625b26ff4b3b8 P 15b1f41f721649dd8a97d314979ac521 [term 1 FOLLOWER]: Refusing update from remote peer 9978d1b39862456881b65ee097849482: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:20:23.991175 4240 raft_consensus.cc:1261] T ebd9c98e971a40b3931625b26ff4b3b8 P f3c3608dbc5146a5affe080bc3253460 [term 1 FOLLOWER]: Refusing update from remote peer 9978d1b39862456881b65ee097849482: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:20:23.991468 4574 consensus_queue.cc:1035] T ebd9c98e971a40b3931625b26ff4b3b8 P 9978d1b39862456881b65ee097849482 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f3c3608dbc5146a5affe080bc3253460" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 45779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:20:23.991580 4578 consensus_queue.cc:1035] T ebd9c98e971a40b3931625b26ff4b3b8 P 9978d1b39862456881b65ee097849482 [LEADER]: Connected to new peer: Peer: permanent_uuid: "15b1f41f721649dd8a97d314979ac521" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 33143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:20:23.993883 4581 mvcc.cc:205] Tried to move back new op lower bound from 7334510690265554944 to 7334510690237685760. Current Snapshot: MvccSnapshot[applied={T|T < 7334510690265554944}]
I20260929 03:20:23.994201 4582 mvcc.cc:205] Tried to move back new op lower bound from 7334510690265554944 to 7334510690237685760. Current Snapshot: MvccSnapshot[applied={T|T < 7334510690265554944}]
I20260929 03:20:23.994382 4583 mvcc.cc:205] Tried to move back new op lower bound from 7334510690265554944 to 7334510690237685760. Current Snapshot: MvccSnapshot[applied={T|T < 7334510690265554944}]
I20260929 03:20:23.996100 4095 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40316:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:20:23.996181 4095 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40316:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:20:23.996862 4095 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 31cae0b535f04bca833ae9ab7318f4b2: Sending DeleteTablet for 3 replicas of tablet ebd9c98e971a40b3931625b26ff4b3b8
I20260929 03:20:23.997179 4220 tablet_service.cc:1514] Processing DeleteTablet for tablet ebd9c98e971a40b3931625b26ff4b3b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:20:23 UTC) from {username='kudu', principal='kudu/127.20.178.126@KRBTEST.COM'} at 127.0.0.1:48090
I20260929 03:20:23.997196 4354 tablet_service.cc:1514] Processing DeleteTablet for tablet ebd9c98e971a40b3931625b26ff4b3b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:20:23 UTC) from {username='kudu', principal='kudu/127.20.178.126@KRBTEST.COM'} at 127.0.0.1:41912
I20260929 03:20:23.997234 4488 tablet_service.cc:1514] Processing DeleteTablet for tablet ebd9c98e971a40b3931625b26ff4b3b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:20:23 UTC) from {username='kudu', principal='kudu/127.20.178.126@KRBTEST.COM'} at 127.0.0.1:36124
I20260929 03:20:23.997270 21193 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuoaHbX/build/release/bin/kudu with pid 4157
W20260929 03:20:24.003679 4079 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.20.178.65:45779 (error 108)
I20260929 03:20:24.003753 4591 tablet_replica.cc:330] T ebd9c98e971a40b3931625b26ff4b3b8 P 9978d1b39862456881b65ee097849482: stopping tablet replica
I20260929 03:20:24.003862 4591 raft_consensus.cc:2229] T ebd9c98e971a40b3931625b26ff4b3b8 P 9978d1b39862456881b65ee097849482 [term 1 LEADER]: Raft consensus shutting down.
I20260929 03:20:24.003991 4591 raft_consensus.cc:2258] T ebd9c98e971a40b3931625b26ff4b3b8 P 9978d1b39862456881b65ee097849482 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:20:24.004170 4564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:20:24.004049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.65:45779 (local address 127.0.0.1:48098)
0929 03:20:24.004097 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0929 03:20:24.004134 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.178.65:45779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260929 03:20:24.004279 4079 catalog_manager.cc:4584] TS f3c3608dbc5146a5affe080bc3253460 (127.20.178.65:45779): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ebd9c98e971a40b3931625b26ff4b3b8: Network error: Client connection negotiation failed: client connection to 127.20.178.65:45779: connect: Connection refused (error 111)
I20260929 03:20:24.004472 4593 tablet_replica.cc:330] T ebd9c98e971a40b3931625b26ff4b3b8 P 15b1f41f721649dd8a97d314979ac521: stopping tablet replica
I20260929 03:20:24.004492 4591 ts_tablet_manager.cc:1905] T ebd9c98e971a40b3931625b26ff4b3b8 P 9978d1b39862456881b65ee097849482: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:20:24.004534 21193 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuoaHbX/build/release/bin/kudu with pid 4291
I20260929 03:20:24.004570 4593 raft_consensus.cc:2229] T ebd9c98e971a40b3931625b26ff4b3b8 P 15b1f41f721649dd8a97d314979ac521 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:20:24.004722 4593 raft_consensus.cc:2258] T ebd9c98e971a40b3931625b26ff4b3b8 P 15b1f41f721649dd8a97d314979ac521 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:20:24.005534 4593 ts_tablet_manager.cc:1905] T ebd9c98e971a40b3931625b26ff4b3b8 P 15b1f41f721649dd8a97d314979ac521: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:20:24.007323 4593 ts_tablet_manager.cc:1918] T ebd9c98e971a40b3931625b26ff4b3b8 P 15b1f41f721649dd8a97d314979ac521: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:20:24.007405 4593 log.cc:1196] T ebd9c98e971a40b3931625b26ff4b3b8 P 15b1f41f721649dd8a97d314979ac521: Deleting WAL directory at /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/minicluster-data/ts-2/wal/wals/ebd9c98e971a40b3931625b26ff4b3b8
I20260929 03:20:24.007695 4593 ts_tablet_manager.cc:1939] T ebd9c98e971a40b3931625b26ff4b3b8 P 15b1f41f721649dd8a97d314979ac521: Deleting consensus metadata
I20260929 03:20:24.008114 4080 catalog_manager.cc:4841] TS 15b1f41f721649dd8a97d314979ac521 (127.20.178.67:33143): tablet ebd9c98e971a40b3931625b26ff4b3b8 (table test-table [id=0ac7856ecacb454a87cad08ba1eb5d91]) successfully deleted
I20260929 03:20:24.010429 21193 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuoaHbX/build/release/bin/kudu with pid 4425
I20260929 03:20:24.010455 4564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:20:24.010325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.66:43451 (local address 127.0.0.1:41940)
0929 03:20:24.010384 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0929 03:20:24.010417 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.178.66:43451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260929 03:20:24.015172 21193 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuoaHbX/build/release/bin/kudu with pid 4063
I20260929 03:20:44.020769 21193 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuoaHbX/build/release/bin/kudu
/tmp/dist-test-taskuoaHbX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.178.126:36419
--webserver_interface=127.20.178.126
--webserver_port=38909
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.126.keytab
--principal=kudu/127.20.178.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.178.84:42735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.178.126:36419
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:20:44.102010 4595 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:20:44.102249 4595 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:20:44.102288 4595 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:20:44.103439 4595 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260929 03:20:44.103499 4595 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260929 03:20:44.103521 4595 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260929 03:20:44.103540 4595 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:20:44.103570 4595 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:20:44.103595 4595 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:20:44.103626 4595 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:20:44.103652 4595 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:20:44.105139 4595 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.178.84:42735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.20.178.126:36419
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.126.keytab
--principal=kudu/127.20.178.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.178.126:36419
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.178.126
--webserver_port=38909
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4595
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:20:44.105424 4595 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:20:44.105650 4595 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:20:44.114315 4602 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:20:44.114395 4604 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:20:44.114342 4601 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:20:44.114509 4595 server_base.cc:1034] running on GCE node
I20260929 03:20:44.114773 4595 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:20:44.115073 4595 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:20:44.116211 4595 hybrid_clock.cc:648] HybridClock initialized: now 1790652044116192 us; error 39 us; skew 500 ppm
Sep 29 03:20:44 dist-test-slave-c9qh krb5kdc[4050](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652044, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:20:44.125468 4595 init.cc:377] Logged in from keytab as kudu/127.20.178.126@KRBTEST.COM (short username kudu)
I20260929 03:20:44.125928 4595 webserver.cc:460] Webserver started at http://127.20.178.126:38909/ using document root <none> and password file <none>
I20260929 03:20:44.126179 4595 fs_manager.cc:359] Metadata directory not provided
I20260929 03:20:44.126253 4595 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:20:44.127610 4595 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 03:20:44.128249 4611 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:20:44.128415 4595 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 03:20:44.128507 4595 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/minicluster-data/master-0/data,/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/minicluster-data/master-0/wal
uuid: "31cae0b535f04bca833ae9ab7318f4b2"
format_stamp: "Formatted at 2026-09-29 03:20:23 on dist-test-slave-c9qh"
server_key: "986112c2b7206bc8770ea147775edf07"
server_key_iv: "c69716b22cda7a2a894ffe04d5ff4e22"
server_key_version: "encryptionkey@0"
I20260929 03:20:44.128995 4595 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:20:44.142740 4595 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:20:44.144747 4595 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:20:44.144897 4595 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:20:44.148824 4595 rpc_server.cc:307] RPC server started. Bound to: 127.20.178.126:36419
I20260929 03:20:44.148862 4663 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.178.126:36419 every 8 connection(s)
I20260929 03:20:44.149219 4595 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/minicluster-data/master-0/data/info.pb
I20260929 03:20:44.150593 4664 sys_catalog.cc:263] Verifying existing consensus state
I20260929 03:20:44.151245 4664 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31cae0b535f04bca833ae9ab7318f4b2: Bootstrap starting.
I20260929 03:20:44.155658 21193 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuoaHbX/build/release/bin/kudu as pid 4595
I20260929 03:20:44.155959 21193 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuoaHbX/build/release/bin/kudu
/tmp/dist-test-taskuoaHbX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.178.65:45779
--local_ip_for_outbound_sockets=127.20.178.65
--tserver_master_addrs=127.20.178.126:36419
--webserver_port=33855
--webserver_interface=127.20.178.65
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.65.keytab
--principal=kudu/127.20.178.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.178.84:42735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 03:20:44.159807 4664 log.cc:824] T 00000000000000000000000000000000 P 31cae0b535f04bca833ae9ab7318f4b2: Log is configured to *not* fsync() on all Append() calls
I20260929 03:20:44.161769 4664 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31cae0b535f04bca833ae9ab7318f4b2: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 03:20:44.162081 4664 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31cae0b535f04bca833ae9ab7318f4b2: Bootstrap complete.
I20260929 03:20:44.164276 4664 raft_consensus.cc:348] T 00000000000000000000000000000000 P 31cae0b535f04bca833ae9ab7318f4b2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31cae0b535f04bca833ae9ab7318f4b2" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 36419 } }
I20260929 03:20:44.164510 4664 raft_consensus.cc:729] T 00000000000000000000000000000000 P 31cae0b535f04bca833ae9ab7318f4b2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31cae0b535f04bca833ae9ab7318f4b2, State: Initialized, Role: FOLLOWER
I20260929 03:20:44.164736 4664 consensus_queue.cc:260] T 00000000000000000000000000000000 P 31cae0b535f04bca833ae9ab7318f4b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31cae0b535f04bca833ae9ab7318f4b2" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 36419 } }
I20260929 03:20:44.164814 4664 raft_consensus.cc:388] T 00000000000000000000000000000000 P 31cae0b535f04bca833ae9ab7318f4b2 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:20:44.164873 4664 raft_consensus.cc:482] T 00000000000000000000000000000000 P 31cae0b535f04bca833ae9ab7318f4b2 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:20:44.164920 4664 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 31cae0b535f04bca833ae9ab7318f4b2 [term 1 FOLLOWER]: Advancing to term 2
I20260929 03:20:44.165887 4664 raft_consensus.cc:504] T 00000000000000000000000000000000 P 31cae0b535f04bca833ae9ab7318f4b2 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31cae0b535f04bca833ae9ab7318f4b2" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 36419 } }
I20260929 03:20:44.166002 4664 leader_election.cc:302] T 00000000000000000000000000000000 P 31cae0b535f04bca833ae9ab7318f4b2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 31cae0b535f04bca833ae9ab7318f4b2; no voters:
I20260929 03:20:44.166181 4664 leader_election.cc:288] T 00000000000000000000000000000000 P 31cae0b535f04bca833ae9ab7318f4b2 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 03:20:44.166213 4668 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 31cae0b535f04bca833ae9ab7318f4b2 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 03:20:44.166402 4668 raft_consensus.cc:686] T 00000000000000000000000000000000 P 31cae0b535f04bca833ae9ab7318f4b2 [term 2 LEADER]: Becoming Leader. State: Replica: 31cae0b535f04bca833ae9ab7318f4b2, State: Running, Role: LEADER
I20260929 03:20:44.166481 4664 sys_catalog.cc:564] T 00000000000000000000000000000000 P 31cae0b535f04bca833ae9ab7318f4b2 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:20:44.166488 4668 consensus_queue.cc:237] T 00000000000000000000000000000000 P 31cae0b535f04bca833ae9ab7318f4b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31cae0b535f04bca833ae9ab7318f4b2" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 36419 } }
I20260929 03:20:44.166764 4670 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31cae0b535f04bca833ae9ab7318f4b2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "31cae0b535f04bca833ae9ab7318f4b2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31cae0b535f04bca833ae9ab7318f4b2" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 36419 } } }
I20260929 03:20:44.166847 4670 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31cae0b535f04bca833ae9ab7318f4b2 [sys.catalog]: This master's current role is: LEADER
I20260929 03:20:44.167076 4669 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31cae0b535f04bca833ae9ab7318f4b2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 31cae0b535f04bca833ae9ab7318f4b2. Latest consensus state: current_term: 2 leader_uuid: "31cae0b535f04bca833ae9ab7318f4b2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31cae0b535f04bca833ae9ab7318f4b2" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 36419 } } }
I20260929 03:20:44.167172 4669 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31cae0b535f04bca833ae9ab7318f4b2 [sys.catalog]: This master's current role is: LEADER
I20260929 03:20:44.167193 4673 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:20:44.168049 4673 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:20:44.168152 4673 catalog_manager.cc:1259] Loaded cluster ID: 590b9fa8fbf44c89a1fd76061dafecc8
I20260929 03:20:44.168190 4673 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:20:44.169162 4673 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:20:44.169345 4673 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 31cae0b535f04bca833ae9ab7318f4b2: Loaded TSK: 0
I20260929 03:20:44.183068 4673 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 31cae0b535f04bca833ae9ab7318f4b2: Generated new TSK 1
I20260929 03:20:44.183200 4673 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 03:20:44.250406 4666 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:20:44.250635 4666 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:20:44.250672 4666 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:20:44.251915 4666 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:20:44.251981 4666 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:20:44.252038 4666 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.178.65
I20260929 03:20:44.253568 4666 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.178.84:42735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.65.keytab
--principal=kudu/127.20.178.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.178.65:45779
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.178.65
--webserver_port=33855
--tserver_master_addrs=127.20.178.126:36419
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4666
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.178.65
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:20:44.253813 4666 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:20:44.254040 4666 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:20:44.258998 4694 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:20:44.259014 4692 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:20:44.259007 4691 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:20:44.259748 4666 server_base.cc:1034] running on GCE node
I20260929 03:20:44.259979 4666 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:20:44.260257 4666 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:20:44.261430 4666 hybrid_clock.cc:648] HybridClock initialized: now 1790652044261412 us; error 35 us; skew 500 ppm
Sep 29 03:20:44 dist-test-slave-c9qh krb5kdc[4050](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652044, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:20:44.263900 4666 init.cc:377] Logged in from keytab as kudu/127.20.178.65@KRBTEST.COM (short username kudu)
I20260929 03:20:44.264379 4666 webserver.cc:460] Webserver started at http://127.20.178.65:33855/ using document root <none> and password file <none>
I20260929 03:20:44.264612 4666 fs_manager.cc:359] Metadata directory not provided
I20260929 03:20:44.264744 4666 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:20:44.266212 4666 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260929 03:20:44.267071 4701 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:20:44.267278 4666 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260929 03:20:44.267373 4666 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/minicluster-data/ts-0/data,/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/minicluster-data/ts-0/wal
uuid: "f3c3608dbc5146a5affe080bc3253460"
format_stamp: "Formatted at 2026-09-29 03:20:23 on dist-test-slave-c9qh"
server_key: "befe8cef71909dd162379b1998d3d44c"
server_key_iv: "611115b899673601adc691edbc0b0d40"
server_key_version: "encryptionkey@0"
I20260929 03:20:44.267863 4666 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:20:44.284042 4666 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:20:44.286720 4666 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:20:44.286878 4666 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:20:44.287148 4666 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:20:44.287683 4708 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260929 03:20:44.289196 4666 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260929 03:20:44.289256 4666 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260929 03:20:44.289294 4666 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260929 03:20:44.290134 4666 ts_tablet_manager.cc:610] Registered 1 tablets
I20260929 03:20:44.290194 4666 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260929 03:20:44.290258 4708 tablet_bootstrap.cc:492] T ebd9c98e971a40b3931625b26ff4b3b8 P f3c3608dbc5146a5affe080bc3253460: Bootstrap starting.
I20260929 03:20:44.297425 4666 rpc_server.cc:307] RPC server started. Bound to: 127.20.178.65:45779
I20260929 03:20:44.297641 4815 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.178.65:45779 every 8 connection(s)
I20260929 03:20:44.297881 4666 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/minicluster-data/ts-0/data/info.pb
I20260929 03:20:44.300892 4708 log.cc:824] T ebd9c98e971a40b3931625b26ff4b3b8 P f3c3608dbc5146a5affe080bc3253460: Log is configured to *not* fsync() on all Append() calls
I20260929 03:20:44.301148 21193 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuoaHbX/build/release/bin/kudu as pid 4666
I20260929 03:20:44.301447 21193 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuoaHbX/build/release/bin/kudu
/tmp/dist-test-taskuoaHbX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.178.66:43451
--local_ip_for_outbound_sockets=127.20.178.66
--tserver_master_addrs=127.20.178.126:36419
--webserver_port=34237
--webserver_interface=127.20.178.66
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.66.keytab
--principal=kudu/127.20.178.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.178.84:42735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 03:20:44.303400 4708 tablet_bootstrap.cc:492] T ebd9c98e971a40b3931625b26ff4b3b8 P f3c3608dbc5146a5affe080bc3253460: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 03:20:44.303728 4708 tablet_bootstrap.cc:492] T ebd9c98e971a40b3931625b26ff4b3b8 P f3c3608dbc5146a5affe080bc3253460: Bootstrap complete.
I20260929 03:20:44.304929 4708 ts_tablet_manager.cc:1397] T ebd9c98e971a40b3931625b26ff4b3b8 P f3c3608dbc5146a5affe080bc3253460: Time spent bootstrapping tablet: real 0.015s user 0.009s sys 0.005s
I20260929 03:20:44.306679 4708 raft_consensus.cc:348] T ebd9c98e971a40b3931625b26ff4b3b8 P f3c3608dbc5146a5affe080bc3253460 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c3608dbc5146a5affe080bc3253460" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 45779 } } peers { permanent_uuid: "15b1f41f721649dd8a97d314979ac521" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 33143 } } peers { permanent_uuid: "9978d1b39862456881b65ee097849482" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 43451 } }
I20260929 03:20:44.307104 4708 raft_consensus.cc:729] T ebd9c98e971a40b3931625b26ff4b3b8 P f3c3608dbc5146a5affe080bc3253460 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3c3608dbc5146a5affe080bc3253460, State: Initialized, Role: FOLLOWER
I20260929 03:20:44.307261 4708 consensus_queue.cc:260] T ebd9c98e971a40b3931625b26ff4b3b8 P f3c3608dbc5146a5affe080bc3253460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c3608dbc5146a5affe080bc3253460" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 45779 } } peers { permanent_uuid: "15b1f41f721649dd8a97d314979ac521" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 33143 } } peers { permanent_uuid: "9978d1b39862456881b65ee097849482" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 43451 } }
I20260929 03:20:44.307915 4708 ts_tablet_manager.cc:1428] T ebd9c98e971a40b3931625b26ff4b3b8 P f3c3608dbc5146a5affe080bc3253460: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
Sep 29 03:20:44 dist-test-slave-c9qh krb5kdc[4050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652044, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.65@KRBTEST.COM for kudu/127.20.178.126@KRBTEST.COM
I20260929 03:20:44.314296 4818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:20:44.298668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:36419 (local address 127.20.178.65:58357)
0929 03:20:44.300772 (+ 2104us) negotiation.cc:107] Waiting for socket to connect
0929 03:20:44.300821 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 03:20:44.302022 (+ 1201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:20:44.302613 (+ 591us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:20:44.302617 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:20:44.302972 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:20:44.303238 (+ 266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:20:44.303244 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:20:44.304881 (+ 1637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:20:44.304882 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:20:44.306108 (+ 1226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:20:44.306110 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:20:44.306940 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:20:44.306941 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:20:44.307070 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:20:44.307698 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:20:44.307708 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:20:44.309577 (+ 1869us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:20:44.312338 (+ 2761us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:20:44.312345 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:20:44.312347 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:20:44.312706 (+ 359us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:20:44.313012 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:20:44.313013 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:20:44.313013 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:20:44.313185 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:20:44.313660 (+ 475us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:20:44.313667 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:20:44.313822 (+ 155us) client_negotiation.cc:769] Sending connection context
0929 03:20:44.313925 (+ 103us) client_negotiation.cc:240] Negotiation successful
0929 03:20:44.314083 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2050,"thread_start_us":100,"threads_started":1}
I20260929 03:20:44.314621 4819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:20:44.298759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.65:58357 (local address 127.20.178.126:36419)
0929 03:20:44.301106 (+ 2347us) server_negotiation.cc:206] Beginning negotiation
0929 03:20:44.301117 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 03:20:44.301164 (+ 47us) server_negotiation.cc:407] Connection header received
0929 03:20:44.302303 (+ 1139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:20:44.302309 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:20:44.302361 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:20:44.302482 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:20:44.303368 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:20:44.304742 (+ 1374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:20:44.306221 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:20:44.306821 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:20:44.307355 (+ 534us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:20:44.310141 (+ 2786us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:20:44.310146 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:20:44.310147 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:20:44.310158 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:20:44.312237 (+ 2079us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:20:44.312793 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:20:44.312796 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:20:44.312796 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:20:44.312919 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:20:44.313276 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:20:44.313277 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:20:44.313278 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:20:44.313574 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:20:44.313660 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0929 03:20:44.314020 (+ 360us) server_negotiation.cc:299] Negotiation successful
0929 03:20:44.314155 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2281,"thread_start_us":87,"threads_started":1}
I20260929 03:20:44.315274 4816 heartbeater.cc:347] Connected to a master server at 127.20.178.126:36419
I20260929 03:20:44.315377 4816 heartbeater.cc:464] Registering TS with master...
I20260929 03:20:44.315534 4816 heartbeater.cc:511] Master 127.20.178.126:36419 requested a full tablet report, sending...
I20260929 03:20:44.316295 4628 ts_manager.cc:194] Registered new tserver with Master: f3c3608dbc5146a5affe080bc3253460 (127.20.178.65:45779)
I20260929 03:20:44.317553 4628 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.178.65@KRBTEST.COM'} at 127.20.178.65:58357
I20260929 03:20:44.318817 4816 heartbeater.cc:503] Master 127.20.178.126:36419 was elected leader, sending a full tablet report...
I20260929 03:20:44.322870 4823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:20:44.317963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.65:45779 (local address 127.0.0.1:39882)
0929 03:20:44.318128 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0929 03:20:44.318161 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0929 03:20:44.318237 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:20:44.319405 (+ 1168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:20:44.319408 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:20:44.319410 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:20:44.319464 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:20:44.319466 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:20:44.320724 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:20:44.320725 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:20:44.321822 (+ 1097us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:20:44.321825 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:20:44.322663 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:20:44.322664 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:20:44.322719 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:20:44.322729 (+ 10us) client_negotiation.cc:769] Sending connection context
0929 03:20:44.322784 (+ 55us) client_negotiation.cc:240] Negotiation successful
0929 03:20:44.322821 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":85,"threads_started":1}
I20260929 03:20:44.322911 4824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:20:44.318199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39882 (local address 127.20.178.65:45779)
0929 03:20:44.319154 (+ 955us) server_negotiation.cc:206] Beginning negotiation
0929 03:20:44.319158 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:20:44.319168 (+ 10us) server_negotiation.cc:407] Connection header received
0929 03:20:44.319201 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:20:44.319203 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:20:44.319218 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:20:44.319358 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:20:44.319533 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:20:44.320564 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:20:44.321907 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:20:44.322577 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:20:44.322657 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:20:44.322706 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0929 03:20:44.322813 (+ 107us) server_negotiation.cc:299] Negotiation successful
0929 03:20:44.322849 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":791,"thread_start_us":63,"threads_started":1}
I20260929 03:20:44.323316 4750 tablet_service.cc:1514] Processing DeleteTablet for tablet ebd9c98e971a40b3931625b26ff4b3b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:20:23 UTC) from {username='kudu', principal='kudu/127.20.178.126@KRBTEST.COM'} at 127.0.0.1:39882
I20260929 03:20:44.323841 4825 tablet_replica.cc:330] stopping tablet replica
I20260929 03:20:44.323962 4825 raft_consensus.cc:2229] T ebd9c98e971a40b3931625b26ff4b3b8 P f3c3608dbc5146a5affe080bc3253460 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:20:44.324038 4825 raft_consensus.cc:2258] T ebd9c98e971a40b3931625b26ff4b3b8 P f3c3608dbc5146a5affe080bc3253460 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:20:44.324411 4825 ts_tablet_manager.cc:1905] T ebd9c98e971a40b3931625b26ff4b3b8 P f3c3608dbc5146a5affe080bc3253460: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:20:44.326922 4825 ts_tablet_manager.cc:1918] T ebd9c98e971a40b3931625b26ff4b3b8 P f3c3608dbc5146a5affe080bc3253460: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:20:44.327015 4825 log.cc:1196] T ebd9c98e971a40b3931625b26ff4b3b8 P f3c3608dbc5146a5affe080bc3253460: Deleting WAL directory at /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/minicluster-data/ts-0/wal/wals/ebd9c98e971a40b3931625b26ff4b3b8
I20260929 03:20:44.327263 4825 ts_tablet_manager.cc:1939] T ebd9c98e971a40b3931625b26ff4b3b8 P f3c3608dbc5146a5affe080bc3253460: Deleting consensus metadata
I20260929 03:20:44.327764 4612 catalog_manager.cc:4841] TS f3c3608dbc5146a5affe080bc3253460 (127.20.178.65:45779): tablet ebd9c98e971a40b3931625b26ff4b3b8 (table test-table [id=0ac7856ecacb454a87cad08ba1eb5d91]) successfully deleted
W20260929 03:20:44.388785 4821 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:20:44.388969 4821 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:20:44.388991 4821 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:20:44.390166 4821 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:20:44.390213 4821 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:20:44.390239 4821 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.178.66
I20260929 03:20:44.391690 4821 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.178.84:42735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.66.keytab
--principal=kudu/127.20.178.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.178.66:43451
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.178.66
--webserver_port=34237
--tserver_master_addrs=127.20.178.126:36419
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4821
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.178.66
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:20:44.391925 4821 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:20:44.392180 4821 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:20:44.397396 4831 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:20:44.397389 4830 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:20:44.397390 4833 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:20:44.397637 4821 server_base.cc:1034] running on GCE node
I20260929 03:20:44.397862 4821 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:20:44.398105 4821 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:20:44.399274 4821 hybrid_clock.cc:648] HybridClock initialized: now 1790652044399257 us; error 33 us; skew 500 ppm
Sep 29 03:20:44 dist-test-slave-c9qh krb5kdc[4050](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652044, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:20:44.401815 4821 init.cc:377] Logged in from keytab as kudu/127.20.178.66@KRBTEST.COM (short username kudu)
I20260929 03:20:44.402297 4821 webserver.cc:460] Webserver started at http://127.20.178.66:34237/ using document root <none> and password file <none>
I20260929 03:20:44.402539 4821 fs_manager.cc:359] Metadata directory not provided
I20260929 03:20:44.402624 4821 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:20:44.403949 4821 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 03:20:44.404762 4840 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:20:44.404935 4821 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 03:20:44.405090 4821 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/minicluster-data/ts-1/data,/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/minicluster-data/ts-1/wal
uuid: "9978d1b39862456881b65ee097849482"
format_stamp: "Formatted at 2026-09-29 03:20:23 on dist-test-slave-c9qh"
server_key: "aaefa21c358dc74638c24d23eda049bd"
server_key_iv: "82497572b79b0a0a6d8302733bc33410"
server_key_version: "encryptionkey@0"
I20260929 03:20:44.405411 4821 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:20:44.413791 4821 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:20:44.415576 4821 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:20:44.415728 4821 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:20:44.415932 4821 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:20:44.416356 4847 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260929 03:20:44.417863 4821 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260929 03:20:44.417928 4821 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260929 03:20:44.417970 4821 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260929 03:20:44.418592 4821 ts_tablet_manager.cc:610] Registered 1 tablets
I20260929 03:20:44.418661 4821 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260929 03:20:44.418777 4847 tablet_bootstrap.cc:492] T ebd9c98e971a40b3931625b26ff4b3b8 P 9978d1b39862456881b65ee097849482: Bootstrap starting.
I20260929 03:20:44.420454 4847 log.cc:824] T ebd9c98e971a40b3931625b26ff4b3b8 P 9978d1b39862456881b65ee097849482: Log is configured to *not* fsync() on all Append() calls
I20260929 03:20:44.423444 4847 tablet_bootstrap.cc:492] T ebd9c98e971a40b3931625b26ff4b3b8 P 9978d1b39862456881b65ee097849482: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 03:20:44.423787 4847 tablet_bootstrap.cc:492] T ebd9c98e971a40b3931625b26ff4b3b8 P 9978d1b39862456881b65ee097849482: Bootstrap complete.
I20260929 03:20:44.423938 4847 ts_tablet_manager.cc:1397] T ebd9c98e971a40b3931625b26ff4b3b8 P 9978d1b39862456881b65ee097849482: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.002s
I20260929 03:20:44.425228 4821 rpc_server.cc:307] RPC server started. Bound to: 127.20.178.66:43451
I20260929 03:20:44.425283 4955 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.178.66:43451 every 8 connection(s)
I20260929 03:20:44.425596 4821 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/minicluster-data/ts-1/data/info.pb
I20260929 03:20:44.425788 4847 raft_consensus.cc:348] T ebd9c98e971a40b3931625b26ff4b3b8 P 9978d1b39862456881b65ee097849482 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c3608dbc5146a5affe080bc3253460" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 45779 } } peers { permanent_uuid: "15b1f41f721649dd8a97d314979ac521" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 33143 } } peers { permanent_uuid: "9978d1b39862456881b65ee097849482" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 43451 } }
I20260929 03:20:44.426172 4847 raft_consensus.cc:729] T ebd9c98e971a40b3931625b26ff4b3b8 P 9978d1b39862456881b65ee097849482 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9978d1b39862456881b65ee097849482, State: Initialized, Role: FOLLOWER
I20260929 03:20:44.426321 4847 consensus_queue.cc:260] T ebd9c98e971a40b3931625b26ff4b3b8 P 9978d1b39862456881b65ee097849482 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c3608dbc5146a5affe080bc3253460" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 45779 } } peers { permanent_uuid: "15b1f41f721649dd8a97d314979ac521" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 33143 } } peers { permanent_uuid: "9978d1b39862456881b65ee097849482" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 43451 } }
I20260929 03:20:44.426581 4847 ts_tablet_manager.cc:1428] T ebd9c98e971a40b3931625b26ff4b3b8 P 9978d1b39862456881b65ee097849482: Time spent starting tablet: real 0.003s user 0.003s sys 0.001s
I20260929 03:20:44.427713 21193 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuoaHbX/build/release/bin/kudu as pid 4821
I20260929 03:20:44.427996 21193 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuoaHbX/build/release/bin/kudu
/tmp/dist-test-taskuoaHbX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.178.67:33143
--local_ip_for_outbound_sockets=127.20.178.67
--tserver_master_addrs=127.20.178.126:36419
--webserver_port=45173
--webserver_interface=127.20.178.67
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.67.keytab
--principal=kudu/127.20.178.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.178.84:42735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 03:20:44 dist-test-slave-c9qh krb5kdc[4050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652044, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.66@KRBTEST.COM for kudu/127.20.178.126@KRBTEST.COM
I20260929 03:20:44.435488 4819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:20:44.426545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.66:46275 (local address 127.20.178.126:36419)
0929 03:20:44.426632 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0929 03:20:44.426634 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:20:44.426711 (+ 77us) server_negotiation.cc:407] Connection header received
0929 03:20:44.426879 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:20:44.426879 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:20:44.426891 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:20:44.426959 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:20:44.427490 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:20:44.428128 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:20:44.429209 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:20:44.429423 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:20:44.429507 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:20:44.432087 (+ 2580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:20:44.432089 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:20:44.432090 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:20:44.432097 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:20:44.433916 (+ 1819us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:20:44.434414 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:20:44.434416 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:20:44.434416 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:20:44.434483 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:20:44.434765 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:20:44.434766 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:20:44.434766 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:20:44.434896 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:20:44.434980 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0929 03:20:44.435285 (+ 305us) server_negotiation.cc:299] Negotiation successful
0929 03:20:44.435388 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 03:20:44.435703 4958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:20:44.426286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:36419 (local address 127.20.178.66:46275)
0929 03:20:44.426463 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0929 03:20:44.426606 (+ 143us) client_negotiation.cc:174] Beginning negotiation
0929 03:20:44.426804 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:20:44.426995 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:20:44.426998 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:20:44.427251 (+ 253us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:20:44.427420 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:20:44.427424 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:20:44.428266 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:20:44.428267 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:20:44.429112 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:20:44.429114 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:20:44.429518 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:20:44.429519 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:20:44.429639 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:20:44.430184 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:20:44.430191 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:20:44.431989 (+ 1798us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:20:44.434032 (+ 2043us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:20:44.434040 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:20:44.434042 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:20:44.434310 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:20:44.434560 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:20:44.434561 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:20:44.434561 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:20:44.434684 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:20:44.434971 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:20:44.434978 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:20:44.435127 (+ 149us) client_negotiation.cc:769] Sending connection context
0929 03:20:44.435264 (+ 137us) client_negotiation.cc:240] Negotiation successful
0929 03:20:44.435437 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":78,"threads_started":1}
I20260929 03:20:44.436195 4956 heartbeater.cc:347] Connected to a master server at 127.20.178.126:36419
I20260929 03:20:44.436313 4956 heartbeater.cc:464] Registering TS with master...
I20260929 03:20:44.436491 4956 heartbeater.cc:511] Master 127.20.178.126:36419 requested a full tablet report, sending...
I20260929 03:20:44.437129 4628 ts_manager.cc:194] Registered new tserver with Master: 9978d1b39862456881b65ee097849482 (127.20.178.66:43451)
I20260929 03:20:44.437740 4628 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.178.66@KRBTEST.COM'} at 127.20.178.66:46275
I20260929 03:20:44.438549 4956 heartbeater.cc:503] Master 127.20.178.126:36419 was elected leader, sending a full tablet report...
Sep 29 03:20:44 dist-test-slave-c9qh krb5kdc[4050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652044, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.126@KRBTEST.COM for kudu/127.20.178.66@KRBTEST.COM
I20260929 03:20:44.446604 4961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:20:44.437548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52670 (local address 127.20.178.66:43451)
0929 03:20:44.437934 (+ 386us) server_negotiation.cc:206] Beginning negotiation
0929 03:20:44.437939 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:20:44.437950 (+ 11us) server_negotiation.cc:407] Connection header received
0929 03:20:44.437992 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:20:44.437994 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:20:44.438016 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:20:44.438152 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:20:44.438698 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:20:44.439673 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:20:44.440694 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:20:44.441088 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:20:44.441152 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:20:44.443339 (+ 2187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:20:44.443346 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:20:44.443347 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:20:44.443355 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:20:44.445231 (+ 1876us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:20:44.445675 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:20:44.445678 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:20:44.445678 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:20:44.445746 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:20:44.445967 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:20:44.445968 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:20:44.445968 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:20:44.446125 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:20:44.446228 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0929 03:20:44.446405 (+ 177us) server_negotiation.cc:299] Negotiation successful
0929 03:20:44.446496 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":351,"thread_start_us":74,"threads_started":1}
I20260929 03:20:44.446604 4823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:20:44.437466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.66:43451 (local address 127.0.0.1:52670)
0929 03:20:44.437520 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0929 03:20:44.437532 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 03:20:44.437570 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:20:44.438179 (+ 609us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:20:44.438181 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:20:44.438511 (+ 330us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:20:44.438553 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:20:44.438555 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:20:44.439749 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:20:44.439750 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:20:44.440561 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:20:44.440563 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:20:44.441179 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:20:44.441180 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:20:44.441248 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:20:44.441746 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:20:44.441751 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:20:44.443241 (+ 1490us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:20:44.445326 (+ 2085us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:20:44.445333 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:20:44.445336 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:20:44.445558 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:20:44.445823 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:20:44.445824 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:20:44.445824 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:20:44.445887 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:20:44.446207 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:20:44.446214 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:20:44.446269 (+ 55us) client_negotiation.cc:769] Sending connection context
0929 03:20:44.446368 (+ 99us) client_negotiation.cc:240] Negotiation successful
0929 03:20:44.446485 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260929 03:20:44.447017 4873 tablet_service.cc:1514] Processing DeleteTablet for tablet ebd9c98e971a40b3931625b26ff4b3b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:20:23 UTC) from {username='kudu', principal='kudu/127.20.178.126@KRBTEST.COM'} at 127.0.0.1:52670
I20260929 03:20:44.447628 4962 tablet_replica.cc:330] stopping tablet replica
I20260929 03:20:44.447744 4962 raft_consensus.cc:2229] T ebd9c98e971a40b3931625b26ff4b3b8 P 9978d1b39862456881b65ee097849482 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:20:44.447809 4962 raft_consensus.cc:2258] T ebd9c98e971a40b3931625b26ff4b3b8 P 9978d1b39862456881b65ee097849482 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:20:44.449244 4962 ts_tablet_manager.cc:1905] T ebd9c98e971a40b3931625b26ff4b3b8 P 9978d1b39862456881b65ee097849482: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:20:44.451462 4962 ts_tablet_manager.cc:1918] T ebd9c98e971a40b3931625b26ff4b3b8 P 9978d1b39862456881b65ee097849482: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:20:44.451535 4962 log.cc:1196] T ebd9c98e971a40b3931625b26ff4b3b8 P 9978d1b39862456881b65ee097849482: Deleting WAL directory at /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/minicluster-data/ts-1/wal/wals/ebd9c98e971a40b3931625b26ff4b3b8
I20260929 03:20:44.451774 4962 ts_tablet_manager.cc:1939] T ebd9c98e971a40b3931625b26ff4b3b8 P 9978d1b39862456881b65ee097849482: Deleting consensus metadata
I20260929 03:20:44.452353 4615 catalog_manager.cc:4841] TS 9978d1b39862456881b65ee097849482 (127.20.178.66:43451): tablet ebd9c98e971a40b3931625b26ff4b3b8 (table test-table [id=0ac7856ecacb454a87cad08ba1eb5d91]) successfully deleted
W20260929 03:20:44.518857 4960 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:20:44.519070 4960 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:20:44.519096 4960 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:20:44.520380 4960 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:20:44.520426 4960 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:20:44.520452 4960 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.178.67
I20260929 03:20:44.522071 4960 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.178.84:42735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.67.keytab
--principal=kudu/127.20.178.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.178.67:33143
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.178.67
--webserver_port=45173
--tserver_master_addrs=127.20.178.126:36419
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4960
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.178.67
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:20:44.522329 4960 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:20:44.522563 4960 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:20:44.527807 4968 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:20:44.527820 4967 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:20:44.527976 4960 server_base.cc:1034] running on GCE node
W20260929 03:20:44.528015 4970 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:20:44.528312 4960 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:20:44.528576 4960 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:20:44.529724 4960 hybrid_clock.cc:648] HybridClock initialized: now 1790652044529713 us; error 31 us; skew 500 ppm
Sep 29 03:20:44 dist-test-slave-c9qh krb5kdc[4050](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652044, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:20:44.532621 4960 init.cc:377] Logged in from keytab as kudu/127.20.178.67@KRBTEST.COM (short username kudu)
I20260929 03:20:44.533205 4960 webserver.cc:460] Webserver started at http://127.20.178.67:45173/ using document root <none> and password file <none>
I20260929 03:20:44.533447 4960 fs_manager.cc:359] Metadata directory not provided
I20260929 03:20:44.533550 4960 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:20:44.534963 4960 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260929 03:20:44.535581 4977 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:20:44.535753 4960 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 03:20:44.535858 4960 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/minicluster-data/ts-2/data,/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/minicluster-data/ts-2/wal
uuid: "15b1f41f721649dd8a97d314979ac521"
format_stamp: "Formatted at 2026-09-29 03:20:23 on dist-test-slave-c9qh"
server_key: "dca81f8dcb62cb70aa55e2579b502690"
server_key_iv: "3565d2f4dcd71b8419b834ddb84a7a6d"
server_key_version: "encryptionkey@0"
I20260929 03:20:44.536206 4960 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:20:44.570680 4960 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:20:44.572935 4960 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:20:44.573166 4960 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:20:44.573431 4960 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:20:44.573781 4960 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:20:44.573841 4960 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:20:44.573880 4960 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:20:44.573927 4960 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:20:44.580374 4960 rpc_server.cc:307] RPC server started. Bound to: 127.20.178.67:33143
I20260929 03:20:44.580423 5090 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.178.67:33143 every 8 connection(s)
I20260929 03:20:44.580866 4960 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/minicluster-data/ts-2/data/info.pb
I20260929 03:20:44.583196 21193 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuoaHbX/build/release/bin/kudu as pid 4960
Sep 29 03:20:44 dist-test-slave-c9qh krb5kdc[4050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652044, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.67@KRBTEST.COM for kudu/127.20.178.126@KRBTEST.COM
I20260929 03:20:44.591377 5095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:20:44.583668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43638 (local address 127.20.178.126:36419)
0929 03:20:44.583799 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0929 03:20:44.583801 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:20:44.583861 (+ 60us) server_negotiation.cc:407] Connection header received
0929 03:20:44.583944 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:20:44.583944 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:20:44.583953 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:20:44.584009 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:20:44.584824 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:20:44.585658 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:20:44.586599 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:20:44.586932 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:20:44.587009 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:20:44.588378 (+ 1369us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:20:44.588379 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:20:44.588380 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:20:44.588386 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:20:44.590224 (+ 1838us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:20:44.590605 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:20:44.590606 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:20:44.590607 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:20:44.590657 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:20:44.590851 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:20:44.590851 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:20:44.590852 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:20:44.590948 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:20:44.591073 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0929 03:20:44.591169 (+ 96us) server_negotiation.cc:299] Negotiation successful
0929 03:20:44.591229 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":55,"threads_started":1}
I20260929 03:20:44.592289 4819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:20:44.581670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.67:49491 (local address 127.20.178.126:36419)
0929 03:20:44.581763 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0929 03:20:44.581765 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:20:44.581993 (+ 228us) server_negotiation.cc:407] Connection header received
0929 03:20:44.582503 (+ 510us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:20:44.582504 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:20:44.582515 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:20:44.582579 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:20:44.583699 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:20:44.584607 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:20:44.586167 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:20:44.586465 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:20:44.586543 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:20:44.589161 (+ 2618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:20:44.589162 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:20:44.589162 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:20:44.589169 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:20:44.590845 (+ 1676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:20:44.591293 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:20:44.591294 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:20:44.591294 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:20:44.591348 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:20:44.591655 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:20:44.591656 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:20:44.591656 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:20:44.591768 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:20:44.591843 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0929 03:20:44.592129 (+ 286us) server_negotiation.cc:299] Negotiation successful
0929 03:20:44.592205 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 03:20:44.592486 5093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:20:44.581668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:36419 (local address 127.20.178.67:49491)
0929 03:20:44.581859 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0929 03:20:44.581902 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 03:20:44.582390 (+ 488us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:20:44.582645 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:20:44.582649 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:20:44.583269 (+ 620us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:20:44.583562 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:20:44.583571 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:20:44.584739 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:20:44.584741 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:20:44.586072 (+ 1331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:20:44.586075 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:20:44.586582 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:20:44.586583 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:20:44.586731 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:20:44.587366 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:20:44.587375 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:20:44.589035 (+ 1660us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:20:44.590903 (+ 1868us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:20:44.590910 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:20:44.590912 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:20:44.591215 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:20:44.591421 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:20:44.591422 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:20:44.591422 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:20:44.591563 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:20:44.591849 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:20:44.591855 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:20:44.591990 (+ 135us) client_negotiation.cc:769] Sending connection context
0929 03:20:44.592103 (+ 113us) client_negotiation.cc:240] Negotiation successful
0929 03:20:44.592256 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":110,"threads_started":1}
I20260929 03:20:44.592937 5091 heartbeater.cc:347] Connected to a master server at 127.20.178.126:36419
I20260929 03:20:44.593029 5091 heartbeater.cc:464] Registering TS with master...
I20260929 03:20:44.593223 5091 heartbeater.cc:511] Master 127.20.178.126:36419 requested a full tablet report, sending...
I20260929 03:20:44.593606 4628 ts_manager.cc:194] Registered new tserver with Master: 15b1f41f721649dd8a97d314979ac521 (127.20.178.67:33143)
I20260929 03:20:44.593966 4628 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.178.67@KRBTEST.COM'} at 127.20.178.67:49491
I20260929 03:20:44.602873 21193 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 03:20:44.605620 4819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:20:44.603259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43640 (local address 127.20.178.126:36419)
0929 03:20:44.603331 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0929 03:20:44.603333 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:20:44.603415 (+ 82us) server_negotiation.cc:407] Connection header received
0929 03:20:44.603463 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:20:44.603464 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:20:44.603494 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:20:44.603556 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:20:44.603773 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:20:44.604224 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:20:44.605050 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:20:44.605229 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:20:44.605282 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:20:44.605414 (+ 132us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:20:44.605454 (+ 40us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0929 03:20:44.605576 (+ 122us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:43640: token expired
Metrics: {"server-negotiator.queue_time_us":51}
W20260929 03:20:44.605690 4819 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:43640: token expired
W20260929 03:20:44.605800 4559 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.20.178.126:36419)
I20260929 03:20:44.611865 4819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:20:44.606074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43652 (local address 127.20.178.126:36419)
0929 03:20:44.606134 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0929 03:20:44.606136 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:20:44.606145 (+ 9us) server_negotiation.cc:407] Connection header received
0929 03:20:44.606199 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:20:44.606200 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:20:44.606209 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:20:44.606258 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:20:44.606938 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:20:44.607599 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:20:44.608396 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:20:44.608593 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:20:44.608717 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:20:44.609855 (+ 1138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:20:44.609856 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:20:44.609856 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:20:44.609862 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:20:44.611011 (+ 1149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:20:44.611317 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:20:44.611318 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:20:44.611318 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:20:44.611366 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:20:44.611507 (+ 141us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:20:44.611507 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:20:44.611507 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:20:44.611586 (+ 79us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:20:44.611652 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0929 03:20:44.611734 (+ 82us) server_negotiation.cc:299] Negotiation successful
0929 03:20:44.611785 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260929 03:20:44.612969 4628 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43652:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 03:20:44.614518 4628 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 03:20:44.616539 4750 tablet_service.cc:1467] Processing CreateTablet for tablet d00aff55fb054f128e16006aaeac20fc (DEFAULT_TABLE table=test-table [id=f9321266222e436bb83c3000c3acc0b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:20:44.616554 4873 tablet_service.cc:1467] Processing CreateTablet for tablet d00aff55fb054f128e16006aaeac20fc (DEFAULT_TABLE table=test-table [id=f9321266222e436bb83c3000c3acc0b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:20:44.616765 4750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d00aff55fb054f128e16006aaeac20fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:20:44.616772 4873 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d00aff55fb054f128e16006aaeac20fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:20:44.618121 4708 tablet_bootstrap.cc:492] T d00aff55fb054f128e16006aaeac20fc P f3c3608dbc5146a5affe080bc3253460: Bootstrap starting.
I20260929 03:20:44.618412 4708 tablet_bootstrap.cc:654] T d00aff55fb054f128e16006aaeac20fc P f3c3608dbc5146a5affe080bc3253460: Neither blocks nor log segments found. Creating new log.
I20260929 03:20:44.618500 4847 tablet_bootstrap.cc:492] T d00aff55fb054f128e16006aaeac20fc P 9978d1b39862456881b65ee097849482: Bootstrap starting.
I20260929 03:20:44.618947 4847 tablet_bootstrap.cc:654] T d00aff55fb054f128e16006aaeac20fc P 9978d1b39862456881b65ee097849482: Neither blocks nor log segments found. Creating new log.
I20260929 03:20:44.619091 4708 tablet_bootstrap.cc:492] T d00aff55fb054f128e16006aaeac20fc P f3c3608dbc5146a5affe080bc3253460: No bootstrap required, opened a new log
I20260929 03:20:44.619159 4708 ts_tablet_manager.cc:1397] T d00aff55fb054f128e16006aaeac20fc P f3c3608dbc5146a5affe080bc3253460: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260929 03:20:44.619269 4708 raft_consensus.cc:348] T d00aff55fb054f128e16006aaeac20fc P f3c3608dbc5146a5affe080bc3253460 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15b1f41f721649dd8a97d314979ac521" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 33143 } } peers { permanent_uuid: "f3c3608dbc5146a5affe080bc3253460" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 45779 } } peers { permanent_uuid: "9978d1b39862456881b65ee097849482" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 43451 } }
I20260929 03:20:44.619350 4708 raft_consensus.cc:374] T d00aff55fb054f128e16006aaeac20fc P f3c3608dbc5146a5affe080bc3253460 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:20:44.619421 4708 raft_consensus.cc:729] T d00aff55fb054f128e16006aaeac20fc P f3c3608dbc5146a5affe080bc3253460 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3c3608dbc5146a5affe080bc3253460, State: Initialized, Role: FOLLOWER
I20260929 03:20:44.619511 4708 consensus_queue.cc:260] T d00aff55fb054f128e16006aaeac20fc P f3c3608dbc5146a5affe080bc3253460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15b1f41f721649dd8a97d314979ac521" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 33143 } } peers { permanent_uuid: "f3c3608dbc5146a5affe080bc3253460" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 45779 } } peers { permanent_uuid: "9978d1b39862456881b65ee097849482" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 43451 } }
I20260929 03:20:44.619621 4708 ts_tablet_manager.cc:1428] T d00aff55fb054f128e16006aaeac20fc P f3c3608dbc5146a5affe080bc3253460: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260929 03:20:44.619623 4847 tablet_bootstrap.cc:492] T d00aff55fb054f128e16006aaeac20fc P 9978d1b39862456881b65ee097849482: No bootstrap required, opened a new log
I20260929 03:20:44.619663 4847 ts_tablet_manager.cc:1397] T d00aff55fb054f128e16006aaeac20fc P 9978d1b39862456881b65ee097849482: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260929 03:20:44.619800 4847 raft_consensus.cc:348] T d00aff55fb054f128e16006aaeac20fc P 9978d1b39862456881b65ee097849482 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15b1f41f721649dd8a97d314979ac521" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 33143 } } peers { permanent_uuid: "f3c3608dbc5146a5affe080bc3253460" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 45779 } } peers { permanent_uuid: "9978d1b39862456881b65ee097849482" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 43451 } }
I20260929 03:20:44.619858 4847 raft_consensus.cc:374] T d00aff55fb054f128e16006aaeac20fc P 9978d1b39862456881b65ee097849482 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:20:44.619908 4847 raft_consensus.cc:729] T d00aff55fb054f128e16006aaeac20fc P 9978d1b39862456881b65ee097849482 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9978d1b39862456881b65ee097849482, State: Initialized, Role: FOLLOWER
I20260929 03:20:44.619952 4847 consensus_queue.cc:260] T d00aff55fb054f128e16006aaeac20fc P 9978d1b39862456881b65ee097849482 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15b1f41f721649dd8a97d314979ac521" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 33143 } } peers { permanent_uuid: "f3c3608dbc5146a5affe080bc3253460" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 45779 } } peers { permanent_uuid: "9978d1b39862456881b65ee097849482" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 43451 } }
I20260929 03:20:44.620045 4847 ts_tablet_manager.cc:1428] T d00aff55fb054f128e16006aaeac20fc P 9978d1b39862456881b65ee097849482: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260929 03:20:44.621816 4823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:20:44.616332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.67:33143 (local address 127.0.0.1:44772)
0929 03:20:44.616367 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0929 03:20:44.616377 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 03:20:44.616411 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:20:44.618677 (+ 2266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:20:44.618677 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:20:44.618680 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:20:44.618720 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:20:44.618721 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:20:44.620082 (+ 1361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:20:44.620083 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:20:44.621060 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:20:44.621063 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:20:44.621661 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:20:44.621662 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:20:44.621702 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:20:44.621705 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 03:20:44.621747 (+ 42us) client_negotiation.cc:240] Negotiation successful
0929 03:20:44.621773 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19}
I20260929 03:20:44.621842 5097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:20:44.618292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44772 (local address 127.20.178.67:33143)
0929 03:20:44.618482 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0929 03:20:44.618487 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:20:44.618498 (+ 11us) server_negotiation.cc:407] Connection header received
0929 03:20:44.618536 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:20:44.618539 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:20:44.618555 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:20:44.618656 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:20:44.618778 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:20:44.619985 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:20:44.621135 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:20:44.621576 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:20:44.621649 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:20:44.621696 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0929 03:20:44.621759 (+ 63us) server_negotiation.cc:299] Negotiation successful
0929 03:20:44.621786 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":118,"threads_started":1}
I20260929 03:20:44.622483 5025 tablet_service.cc:1467] Processing CreateTablet for tablet d00aff55fb054f128e16006aaeac20fc (DEFAULT_TABLE table=test-table [id=f9321266222e436bb83c3000c3acc0b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:20:44.622742 5025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d00aff55fb054f128e16006aaeac20fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:20:44.625448 5100 tablet_bootstrap.cc:492] T d00aff55fb054f128e16006aaeac20fc P 15b1f41f721649dd8a97d314979ac521: Bootstrap starting.
I20260929 03:20:44.626142 5100 tablet_bootstrap.cc:654] T d00aff55fb054f128e16006aaeac20fc P 15b1f41f721649dd8a97d314979ac521: Neither blocks nor log segments found. Creating new log.
I20260929 03:20:44.626627 5100 log.cc:824] T d00aff55fb054f128e16006aaeac20fc P 15b1f41f721649dd8a97d314979ac521: Log is configured to *not* fsync() on all Append() calls
I20260929 03:20:44.627327 5100 tablet_bootstrap.cc:492] T d00aff55fb054f128e16006aaeac20fc P 15b1f41f721649dd8a97d314979ac521: No bootstrap required, opened a new log
I20260929 03:20:44.627414 5100 ts_tablet_manager.cc:1397] T d00aff55fb054f128e16006aaeac20fc P 15b1f41f721649dd8a97d314979ac521: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 03:20:44.628774 5100 raft_consensus.cc:348] T d00aff55fb054f128e16006aaeac20fc P 15b1f41f721649dd8a97d314979ac521 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15b1f41f721649dd8a97d314979ac521" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 33143 } } peers { permanent_uuid: "f3c3608dbc5146a5affe080bc3253460" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 45779 } } peers { permanent_uuid: "9978d1b39862456881b65ee097849482" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 43451 } }
I20260929 03:20:44.628929 5100 raft_consensus.cc:374] T d00aff55fb054f128e16006aaeac20fc P 15b1f41f721649dd8a97d314979ac521 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:20:44.628973 5100 raft_consensus.cc:729] T d00aff55fb054f128e16006aaeac20fc P 15b1f41f721649dd8a97d314979ac521 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15b1f41f721649dd8a97d314979ac521, State: Initialized, Role: FOLLOWER
I20260929 03:20:44.629128 5100 consensus_queue.cc:260] T d00aff55fb054f128e16006aaeac20fc P 15b1f41f721649dd8a97d314979ac521 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15b1f41f721649dd8a97d314979ac521" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 33143 } } peers { permanent_uuid: "f3c3608dbc5146a5affe080bc3253460" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 45779 } } peers { permanent_uuid: "9978d1b39862456881b65ee097849482" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 43451 } }
I20260929 03:20:44.629415 5100 ts_tablet_manager.cc:1428] T d00aff55fb054f128e16006aaeac20fc P 15b1f41f721649dd8a97d314979ac521: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260929 03:20:44.629432 5091 heartbeater.cc:503] Master 127.20.178.126:36419 was elected leader, sending a full tablet report...
I20260929 03:20:44.632681 4822 raft_consensus.cc:482] T d00aff55fb054f128e16006aaeac20fc P f3c3608dbc5146a5affe080bc3253460 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:20:44.632793 4822 raft_consensus.cc:504] T d00aff55fb054f128e16006aaeac20fc P f3c3608dbc5146a5affe080bc3253460 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15b1f41f721649dd8a97d314979ac521" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 33143 } } peers { permanent_uuid: "f3c3608dbc5146a5affe080bc3253460" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 45779 } } peers { permanent_uuid: "9978d1b39862456881b65ee097849482" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 43451 } }
I20260929 03:20:44.633157 4822 leader_election.cc:288] T d00aff55fb054f128e16006aaeac20fc P f3c3608dbc5146a5affe080bc3253460 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 15b1f41f721649dd8a97d314979ac521 (127.20.178.67:33143), 9978d1b39862456881b65ee097849482 (127.20.178.66:43451)
I20260929 03:20:44.636202 5097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:20:44.633316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.65:45343 (local address 127.20.178.67:33143)
0929 03:20:44.633363 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0929 03:20:44.633365 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:20:44.633477 (+ 112us) server_negotiation.cc:407] Connection header received
0929 03:20:44.633503 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:20:44.633504 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:20:44.633513 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:20:44.633578 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:20:44.633982 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:20:44.634542 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:20:44.635490 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:20:44.635858 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:20:44.635938 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:20:44.635946 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0929 03:20:44.636113 (+ 167us) server_negotiation.cc:299] Negotiation successful
0929 03:20:44.636147 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260929 03:20:44.636204 4818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:20:44.633276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.67:33143 (local address 127.20.178.65:45343)
0929 03:20:44.633364 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0929 03:20:44.633377 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 03:20:44.633431 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:20:44.633599 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:20:44.633600 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:20:44.633601 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:20:44.633905 (+ 304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:20:44.633907 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:20:44.634625 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:20:44.634626 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:20:44.635390 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:20:44.635392 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:20:44.635936 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:20:44.635936 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:20:44.636010 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:20:44.636013 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 03:20:44.636078 (+ 65us) client_negotiation.cc:240] Negotiation successful
0929 03:20:44.636119 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":70,"mutex_wait_us":27}
I20260929 03:20:44.636278 4961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:20:44.633637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.65:46979 (local address 127.20.178.66:43451)
0929 03:20:44.633697 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0929 03:20:44.633699 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:20:44.633706 (+ 7us) server_negotiation.cc:407] Connection header received
0929 03:20:44.633735 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:20:44.633736 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:20:44.633751 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:20:44.633822 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:20:44.633977 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:20:44.634551 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:20:44.635487 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:20:44.635903 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:20:44.635988 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:20:44.636049 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0929 03:20:44.636165 (+ 116us) server_negotiation.cc:299] Negotiation successful
0929 03:20:44.636203 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260929 03:20:44.636418 5103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:20:44.633316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.66:43451 (local address 127.20.178.65:46979)
0929 03:20:44.633467 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0929 03:20:44.633477 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 03:20:44.633527 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:20:44.633827 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:20:44.633828 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:20:44.633830 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:20:44.633890 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:20:44.633892 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:20:44.634637 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:20:44.634638 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:20:44.635400 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:20:44.635402 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:20:44.636009 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:20:44.636010 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:20:44.636087 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:20:44.636090 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 03:20:44.636158 (+ 68us) client_negotiation.cc:240] Negotiation successful
0929 03:20:44.636201 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"mutex_wait_us":1,"thread_start_us":81,"threads_started":1}
I20260929 03:20:44.636726 5045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d00aff55fb054f128e16006aaeac20fc" candidate_uuid: "f3c3608dbc5146a5affe080bc3253460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15b1f41f721649dd8a97d314979ac521" is_pre_election: true
I20260929 03:20:44.636861 5045 raft_consensus.cc:2454] T d00aff55fb054f128e16006aaeac20fc P 15b1f41f721649dd8a97d314979ac521 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f3c3608dbc5146a5affe080bc3253460 in term 0.
I20260929 03:20:44.636797 4891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d00aff55fb054f128e16006aaeac20fc" candidate_uuid: "f3c3608dbc5146a5affe080bc3253460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9978d1b39862456881b65ee097849482" is_pre_election: true
I20260929 03:20:44.636943 4891 raft_consensus.cc:2454] T d00aff55fb054f128e16006aaeac20fc P 9978d1b39862456881b65ee097849482 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f3c3608dbc5146a5affe080bc3253460 in term 0.
I20260929 03:20:44.637069 4703 leader_election.cc:302] T d00aff55fb054f128e16006aaeac20fc P f3c3608dbc5146a5affe080bc3253460 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15b1f41f721649dd8a97d314979ac521, f3c3608dbc5146a5affe080bc3253460; no voters:
I20260929 03:20:44.637207 4822 raft_consensus.cc:2781] T d00aff55fb054f128e16006aaeac20fc P f3c3608dbc5146a5affe080bc3253460 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 03:20:44.637267 4822 raft_consensus.cc:482] T d00aff55fb054f128e16006aaeac20fc P f3c3608dbc5146a5affe080bc3253460 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 03:20:44.637300 4822 raft_consensus.cc:3037] T d00aff55fb054f128e16006aaeac20fc P f3c3608dbc5146a5affe080bc3253460 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:20:44.637931 4822 raft_consensus.cc:504] T d00aff55fb054f128e16006aaeac20fc P f3c3608dbc5146a5affe080bc3253460 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15b1f41f721649dd8a97d314979ac521" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 33143 } } peers { permanent_uuid: "f3c3608dbc5146a5affe080bc3253460" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 45779 } } peers { permanent_uuid: "9978d1b39862456881b65ee097849482" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 43451 } }
I20260929 03:20:44.638087 4822 leader_election.cc:288] T d00aff55fb054f128e16006aaeac20fc P f3c3608dbc5146a5affe080bc3253460 [CANDIDATE]: Term 1 election: Requested vote from peers 15b1f41f721649dd8a97d314979ac521 (127.20.178.67:33143), 9978d1b39862456881b65ee097849482 (127.20.178.66:43451)
I20260929 03:20:44.638230 4891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d00aff55fb054f128e16006aaeac20fc" candidate_uuid: "f3c3608dbc5146a5affe080bc3253460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9978d1b39862456881b65ee097849482"
I20260929 03:20:44.638311 4891 raft_consensus.cc:3037] T d00aff55fb054f128e16006aaeac20fc P 9978d1b39862456881b65ee097849482 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:20:44.639076 4891 raft_consensus.cc:2454] T d00aff55fb054f128e16006aaeac20fc P 9978d1b39862456881b65ee097849482 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f3c3608dbc5146a5affe080bc3253460 in term 1.
I20260929 03:20:44.639207 5045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d00aff55fb054f128e16006aaeac20fc" candidate_uuid: "f3c3608dbc5146a5affe080bc3253460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15b1f41f721649dd8a97d314979ac521"
I20260929 03:20:44.639238 4705 leader_election.cc:302] T d00aff55fb054f128e16006aaeac20fc P f3c3608dbc5146a5affe080bc3253460 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9978d1b39862456881b65ee097849482, f3c3608dbc5146a5affe080bc3253460; no voters:
I20260929 03:20:44.639277 5045 raft_consensus.cc:3037] T d00aff55fb054f128e16006aaeac20fc P 15b1f41f721649dd8a97d314979ac521 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:20:44.639317 4822 raft_consensus.cc:2781] T d00aff55fb054f128e16006aaeac20fc P f3c3608dbc5146a5affe080bc3253460 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:20:44.639447 4822 raft_consensus.cc:686] T d00aff55fb054f128e16006aaeac20fc P f3c3608dbc5146a5affe080bc3253460 [term 1 LEADER]: Becoming Leader. State: Replica: f3c3608dbc5146a5affe080bc3253460, State: Running, Role: LEADER
I20260929 03:20:44.639513 4822 consensus_queue.cc:237] T d00aff55fb054f128e16006aaeac20fc P f3c3608dbc5146a5affe080bc3253460 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15b1f41f721649dd8a97d314979ac521" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 33143 } } peers { permanent_uuid: "f3c3608dbc5146a5affe080bc3253460" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 45779 } } peers { permanent_uuid: "9978d1b39862456881b65ee097849482" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 43451 } }
I20260929 03:20:44.640081 5045 raft_consensus.cc:2454] T d00aff55fb054f128e16006aaeac20fc P 15b1f41f721649dd8a97d314979ac521 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f3c3608dbc5146a5affe080bc3253460 in term 1.
I20260929 03:20:44.640197 4627 catalog_manager.cc:5495] T d00aff55fb054f128e16006aaeac20fc P f3c3608dbc5146a5affe080bc3253460 reported cstate change: term changed from 0 to 1, leader changed from <none> to f3c3608dbc5146a5affe080bc3253460 (127.20.178.65). New cstate: current_term: 1 leader_uuid: "f3c3608dbc5146a5affe080bc3253460" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15b1f41f721649dd8a97d314979ac521" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 33143 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f3c3608dbc5146a5affe080bc3253460" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 45779 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9978d1b39862456881b65ee097849482" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 43451 } health_report { overall_health: UNKNOWN } } }
I20260929 03:20:44.649279 4824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:20:44.647250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39884 (local address 127.20.178.65:45779)
0929 03:20:44.647311 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0929 03:20:44.647313 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:20:44.647347 (+ 34us) server_negotiation.cc:407] Connection header received
0929 03:20:44.647371 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:20:44.647372 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:20:44.647385 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:20:44.647434 (+ 49us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:20:44.647511 (+ 77us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:20:44.647947 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:20:44.648619 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:20:44.648850 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:20:44.648902 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:20:44.649037 (+ 135us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:20:44.649124 (+ 87us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:20:44.649177 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0929 03:20:44.649217 (+ 40us) server_negotiation.cc:299] Negotiation successful
0929 03:20:44.649234 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260929 03:20:44.650668 5045 raft_consensus.cc:1261] T d00aff55fb054f128e16006aaeac20fc P 15b1f41f721649dd8a97d314979ac521 [term 1 FOLLOWER]: Refusing update from remote peer f3c3608dbc5146a5affe080bc3253460: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:20:44.650686 4891 raft_consensus.cc:1261] T d00aff55fb054f128e16006aaeac20fc P 9978d1b39862456881b65ee097849482 [term 1 FOLLOWER]: Refusing update from remote peer f3c3608dbc5146a5affe080bc3253460: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:20:44.650976 5104 consensus_queue.cc:1035] T d00aff55fb054f128e16006aaeac20fc P f3c3608dbc5146a5affe080bc3253460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "15b1f41f721649dd8a97d314979ac521" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 33143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:20:44.651106 4822 consensus_queue.cc:1035] T d00aff55fb054f128e16006aaeac20fc P f3c3608dbc5146a5affe080bc3253460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9978d1b39862456881b65ee097849482" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 43451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:20:44.653169 5107 mvcc.cc:205] Tried to move back new op lower bound from 7334510774887047168 to 7334510774843912192. Current Snapshot: MvccSnapshot[applied={T|T < 7334510774887047168}]
I20260929 03:20:44.653476 5108 mvcc.cc:205] Tried to move back new op lower bound from 7334510774887047168 to 7334510774843912192. Current Snapshot: MvccSnapshot[applied={T|T < 7334510774887047168}]
I20260929 03:20:44.653766 5109 mvcc.cc:205] Tried to move back new op lower bound from 7334510774887047168 to 7334510774843912192. Current Snapshot: MvccSnapshot[applied={T|T < 7334510774887047168}]
I20260929 03:20:44.655084 4627 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43652:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:20:44.655175 4627 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43652:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:20:44.655855 4627 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 31cae0b535f04bca833ae9ab7318f4b2: Sending DeleteTablet for 3 replicas of tablet d00aff55fb054f128e16006aaeac20fc
I20260929 03:20:44.656143 4750 tablet_service.cc:1514] Processing DeleteTablet for tablet d00aff55fb054f128e16006aaeac20fc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:20:44 UTC) from {username='kudu', principal='kudu/127.20.178.126@KRBTEST.COM'} at 127.0.0.1:39882
I20260929 03:20:44.656157 5025 tablet_service.cc:1514] Processing DeleteTablet for tablet d00aff55fb054f128e16006aaeac20fc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:20:44 UTC) from {username='kudu', principal='kudu/127.20.178.126@KRBTEST.COM'} at 127.0.0.1:44772
I20260929 03:20:44.656167 4873 tablet_service.cc:1514] Processing DeleteTablet for tablet d00aff55fb054f128e16006aaeac20fc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:20:44 UTC) from {username='kudu', principal='kudu/127.20.178.126@KRBTEST.COM'} at 127.0.0.1:52670
I20260929 03:20:44.656185 21193 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuoaHbX/build/release/bin/kudu with pid 4666
I20260929 03:20:44.656265 4825 tablet_replica.cc:330] T d00aff55fb054f128e16006aaeac20fc P f3c3608dbc5146a5affe080bc3253460: stopping tablet replica
I20260929 03:20:44.656266 4962 tablet_replica.cc:330] T d00aff55fb054f128e16006aaeac20fc P 9978d1b39862456881b65ee097849482: stopping tablet replica
I20260929 03:20:44.656317 4825 raft_consensus.cc:2229] T d00aff55fb054f128e16006aaeac20fc P f3c3608dbc5146a5affe080bc3253460 [term 1 LEADER]: Raft consensus shutting down.
I20260929 03:20:44.656323 4962 raft_consensus.cc:2229] T d00aff55fb054f128e16006aaeac20fc P 9978d1b39862456881b65ee097849482 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:20:44.656364 4962 raft_consensus.cc:2258] T d00aff55fb054f128e16006aaeac20fc P 9978d1b39862456881b65ee097849482 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:20:44.656391 4825 raft_consensus.cc:2258] T d00aff55fb054f128e16006aaeac20fc P f3c3608dbc5146a5affe080bc3253460 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:20:44.656873 5116 tablet_replica.cc:330] T d00aff55fb054f128e16006aaeac20fc P 15b1f41f721649dd8a97d314979ac521: stopping tablet replica
I20260929 03:20:44.656965 5116 raft_consensus.cc:2229] T d00aff55fb054f128e16006aaeac20fc P 15b1f41f721649dd8a97d314979ac521 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:20:44.656991 4962 ts_tablet_manager.cc:1905] T d00aff55fb054f128e16006aaeac20fc P 9978d1b39862456881b65ee097849482: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:20:44.657060 5116 raft_consensus.cc:2258] T d00aff55fb054f128e16006aaeac20fc P 15b1f41f721649dd8a97d314979ac521 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:20:44.657471 5116 ts_tablet_manager.cc:1905] T d00aff55fb054f128e16006aaeac20fc P 15b1f41f721649dd8a97d314979ac521: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:20:44.658747 4962 ts_tablet_manager.cc:1918] T d00aff55fb054f128e16006aaeac20fc P 9978d1b39862456881b65ee097849482: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:20:44.658828 4962 log.cc:1196] T d00aff55fb054f128e16006aaeac20fc P 9978d1b39862456881b65ee097849482: Deleting WAL directory at /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/minicluster-data/ts-1/wal/wals/d00aff55fb054f128e16006aaeac20fc
I20260929 03:20:44.659047 4962 ts_tablet_manager.cc:1939] T d00aff55fb054f128e16006aaeac20fc P 9978d1b39862456881b65ee097849482: Deleting consensus metadata
I20260929 03:20:44.659228 5116 ts_tablet_manager.cc:1918] T d00aff55fb054f128e16006aaeac20fc P 15b1f41f721649dd8a97d314979ac521: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:20:44.659305 5116 log.cc:1196] T d00aff55fb054f128e16006aaeac20fc P 15b1f41f721649dd8a97d314979ac521: Deleting WAL directory at /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/minicluster-data/ts-2/wal/wals/d00aff55fb054f128e16006aaeac20fc
I20260929 03:20:44.659394 4615 catalog_manager.cc:4841] TS 9978d1b39862456881b65ee097849482 (127.20.178.66:43451): tablet d00aff55fb054f128e16006aaeac20fc (table test-table [id=f9321266222e436bb83c3000c3acc0b4]) successfully deleted
I20260929 03:20:44.659606 5116 ts_tablet_manager.cc:1939] T d00aff55fb054f128e16006aaeac20fc P 15b1f41f721649dd8a97d314979ac521: Deleting consensus metadata
I20260929 03:20:44.659987 4613 catalog_manager.cc:4841] TS 15b1f41f721649dd8a97d314979ac521 (127.20.178.67:33143): tablet d00aff55fb054f128e16006aaeac20fc (table test-table [id=f9321266222e436bb83c3000c3acc0b4]) successfully deleted
W20260929 03:20:44.662477 4612 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.20.178.65:45779 (error 108)
I20260929 03:20:44.662741 21193 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuoaHbX/build/release/bin/kudu with pid 4821
I20260929 03:20:44.662927 4823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:20:44.662811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.65:45779 (local address 127.0.0.1:39896)
0929 03:20:44.662858 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0929 03:20:44.662893 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.178.65:45779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260929 03:20:44.663621 4612 catalog_manager.cc:4584] TS f3c3608dbc5146a5affe080bc3253460 (127.20.178.65:45779): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d00aff55fb054f128e16006aaeac20fc: Network error: Client connection negotiation failed: client connection to 127.20.178.65:45779: connect: Connection refused (error 111)
I20260929 03:20:44.667883 21193 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuoaHbX/build/release/bin/kudu with pid 4960
I20260929 03:20:44.672257 21193 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuoaHbX/build/release/bin/kudu with pid 4595
I20260929 03:21:04.677865 21193 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuoaHbX/build/release/bin/kudu
/tmp/dist-test-taskuoaHbX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.178.126:36419
--webserver_interface=127.20.178.126
--webserver_port=38909
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.126.keytab
--principal=kudu/127.20.178.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.178.84:42735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.178.126:36419
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:21:04.758988 5118 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:21:04.759236 5118 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:21:04.759338 5118 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:21:04.760563 5118 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260929 03:21:04.760602 5118 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260929 03:21:04.760615 5118 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260929 03:21:04.760663 5118 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:21:04.760690 5118 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:21:04.760713 5118 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:21:04.760746 5118 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:21:04.760759 5118 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:21:04.762216 5118 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.178.84:42735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.20.178.126:36419
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.126.keytab
--principal=kudu/127.20.178.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.178.126:36419
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.178.126
--webserver_port=38909
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5118
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:21:04.762472 5118 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:21:04.762660 5118 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:21:04.767524 5123 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:21:04.767482 5126 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:21:04.767500 5124 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:21:04.767715 5118 server_base.cc:1034] running on GCE node
I20260929 03:21:04.767962 5118 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:21:04.768246 5118 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:21:04.769410 5118 hybrid_clock.cc:648] HybridClock initialized: now 1790652064769405 us; error 38 us; skew 500 ppm
Sep 29 03:21:04 dist-test-slave-c9qh krb5kdc[4050](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652064, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:21:04.777396 5118 init.cc:377] Logged in from keytab as kudu/127.20.178.126@KRBTEST.COM (short username kudu)
I20260929 03:21:04.777890 5118 webserver.cc:460] Webserver started at http://127.20.178.126:38909/ using document root <none> and password file <none>
I20260929 03:21:04.778118 5118 fs_manager.cc:359] Metadata directory not provided
I20260929 03:21:04.778187 5118 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:21:04.779520 5118 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 03:21:04.780181 5133 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:21:04.780364 5118 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 03:21:04.780464 5118 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/minicluster-data/master-0/data,/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/minicluster-data/master-0/wal
uuid: "31cae0b535f04bca833ae9ab7318f4b2"
format_stamp: "Formatted at 2026-09-29 03:20:23 on dist-test-slave-c9qh"
server_key: "986112c2b7206bc8770ea147775edf07"
server_key_iv: "c69716b22cda7a2a894ffe04d5ff4e22"
server_key_version: "encryptionkey@0"
I20260929 03:21:04.780925 5118 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:21:04.800235 5118 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:21:04.802791 5118 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:21:04.802958 5118 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:21:04.806958 5118 rpc_server.cc:307] RPC server started. Bound to: 127.20.178.126:36419
I20260929 03:21:04.806993 5185 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.178.126:36419 every 8 connection(s)
I20260929 03:21:04.807395 5118 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/minicluster-data/master-0/data/info.pb
I20260929 03:21:04.808977 5186 sys_catalog.cc:263] Verifying existing consensus state
I20260929 03:21:04.809636 5186 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31cae0b535f04bca833ae9ab7318f4b2: Bootstrap starting.
I20260929 03:21:04.813227 21193 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuoaHbX/build/release/bin/kudu as pid 5118
I20260929 03:21:04.813571 21193 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuoaHbX/build/release/bin/kudu
/tmp/dist-test-taskuoaHbX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.178.65:45779
--local_ip_for_outbound_sockets=127.20.178.65
--tserver_master_addrs=127.20.178.126:36419
--webserver_port=33855
--webserver_interface=127.20.178.65
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.65.keytab
--principal=kudu/127.20.178.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.178.84:42735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 03:21:04.818897 5186 log.cc:824] T 00000000000000000000000000000000 P 31cae0b535f04bca833ae9ab7318f4b2: Log is configured to *not* fsync() on all Append() calls
I20260929 03:21:04.822685 5186 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31cae0b535f04bca833ae9ab7318f4b2: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 03:21:04.823076 5186 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31cae0b535f04bca833ae9ab7318f4b2: Bootstrap complete.
I20260929 03:21:04.826403 5186 raft_consensus.cc:348] T 00000000000000000000000000000000 P 31cae0b535f04bca833ae9ab7318f4b2 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31cae0b535f04bca833ae9ab7318f4b2" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 36419 } }
I20260929 03:21:04.826714 5186 raft_consensus.cc:729] T 00000000000000000000000000000000 P 31cae0b535f04bca833ae9ab7318f4b2 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31cae0b535f04bca833ae9ab7318f4b2, State: Initialized, Role: FOLLOWER
I20260929 03:21:04.826879 5186 consensus_queue.cc:260] T 00000000000000000000000000000000 P 31cae0b535f04bca833ae9ab7318f4b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31cae0b535f04bca833ae9ab7318f4b2" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 36419 } }
I20260929 03:21:04.826942 5186 raft_consensus.cc:388] T 00000000000000000000000000000000 P 31cae0b535f04bca833ae9ab7318f4b2 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:21:04.826978 5186 raft_consensus.cc:482] T 00000000000000000000000000000000 P 31cae0b535f04bca833ae9ab7318f4b2 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:21:04.827023 5186 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 31cae0b535f04bca833ae9ab7318f4b2 [term 2 FOLLOWER]: Advancing to term 3
I20260929 03:21:04.828357 5186 raft_consensus.cc:504] T 00000000000000000000000000000000 P 31cae0b535f04bca833ae9ab7318f4b2 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31cae0b535f04bca833ae9ab7318f4b2" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 36419 } }
I20260929 03:21:04.828466 5186 leader_election.cc:302] T 00000000000000000000000000000000 P 31cae0b535f04bca833ae9ab7318f4b2 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 31cae0b535f04bca833ae9ab7318f4b2; no voters:
I20260929 03:21:04.828677 5186 leader_election.cc:288] T 00000000000000000000000000000000 P 31cae0b535f04bca833ae9ab7318f4b2 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260929 03:21:04.828733 5190 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 31cae0b535f04bca833ae9ab7318f4b2 [term 3 FOLLOWER]: Leader election won for term 3
I20260929 03:21:04.828922 5190 raft_consensus.cc:686] T 00000000000000000000000000000000 P 31cae0b535f04bca833ae9ab7318f4b2 [term 3 LEADER]: Becoming Leader. State: Replica: 31cae0b535f04bca833ae9ab7318f4b2, State: Running, Role: LEADER
I20260929 03:21:04.828925 5186 sys_catalog.cc:564] T 00000000000000000000000000000000 P 31cae0b535f04bca833ae9ab7318f4b2 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:21:04.829176 5190 consensus_queue.cc:237] T 00000000000000000000000000000000 P 31cae0b535f04bca833ae9ab7318f4b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31cae0b535f04bca833ae9ab7318f4b2" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 36419 } }
I20260929 03:21:04.829438 5191 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31cae0b535f04bca833ae9ab7318f4b2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "31cae0b535f04bca833ae9ab7318f4b2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31cae0b535f04bca833ae9ab7318f4b2" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 36419 } } }
I20260929 03:21:04.829519 5191 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31cae0b535f04bca833ae9ab7318f4b2 [sys.catalog]: This master's current role is: LEADER
I20260929 03:21:04.829680 5192 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31cae0b535f04bca833ae9ab7318f4b2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 31cae0b535f04bca833ae9ab7318f4b2. Latest consensus state: current_term: 3 leader_uuid: "31cae0b535f04bca833ae9ab7318f4b2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31cae0b535f04bca833ae9ab7318f4b2" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 36419 } } }
I20260929 03:21:04.829733 5192 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31cae0b535f04bca833ae9ab7318f4b2 [sys.catalog]: This master's current role is: LEADER
I20260929 03:21:04.829946 5199 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:21:04.830766 5199 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:21:04.830878 5199 catalog_manager.cc:1259] Loaded cluster ID: 590b9fa8fbf44c89a1fd76061dafecc8
I20260929 03:21:04.830914 5199 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:21:04.832379 5199 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:21:04.832562 5199 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 31cae0b535f04bca833ae9ab7318f4b2: Loaded TSK: 0
I20260929 03:21:04.832604 5199 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 31cae0b535f04bca833ae9ab7318f4b2: Loaded TSK: 1
F20260929 03:21:04.832876 5199 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fe86c84c9ad google::LogMessage::Fail() at ??:0
@ 0x7fe86c84f88a google::LogMessage::SendToLog() at ??:0
@ 0x7fe86c84c4de google::LogMessage::Flush() at ??:0
@ 0x7fe86c84da19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fe86efb9104 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fe86efb6d41 kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7fe8747aa1cd kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7fe8747aa48e kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7fe8747aa594 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7fe874770c48 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7fe87477aad0 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fe86e71a0d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fe86e70f954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fe86f5736db start_thread at ??:0
@ 0x7fe86b34a61f clone at ??:0
W20260929 03:21:04.916030 5188 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:21:04.916296 5188 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:21:04.916335 5188 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:21:04.917604 5188 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:21:04.917670 5188 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:21:04.917716 5188 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.178.65
I20260929 03:21:04.919257 5188 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.178.84:42735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.65.keytab
--principal=kudu/127.20.178.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.178.65:45779
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.178.65
--webserver_port=33855
--tserver_master_addrs=127.20.178.126:36419
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5188
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.178.65
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:21:04.919515 5188 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:21:04.919749 5188 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:21:04.924736 5227 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:21:04.924893 5188 server_base.cc:1034] running on GCE node
W20260929 03:21:04.925026 5230 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:21:04.925031 5228 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:21:04.925326 5188 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:21:04.925606 5188 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:21:04.926800 5188 hybrid_clock.cc:648] HybridClock initialized: now 1790652064926784 us; error 29 us; skew 500 ppm
Sep 29 03:21:04 dist-test-slave-c9qh krb5kdc[4050](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652064, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:21:04.929912 5188 init.cc:377] Logged in from keytab as kudu/127.20.178.65@KRBTEST.COM (short username kudu)
I20260929 03:21:04.930438 5188 webserver.cc:460] Webserver started at http://127.20.178.65:33855/ using document root <none> and password file <none>
I20260929 03:21:04.930702 5188 fs_manager.cc:359] Metadata directory not provided
I20260929 03:21:04.930765 5188 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:21:04.932502 5188 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 03:21:04.933332 5237 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:21:04.933547 5188 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260929 03:21:04.933637 5188 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/minicluster-data/ts-0/data,/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/minicluster-data/ts-0/wal
uuid: "f3c3608dbc5146a5affe080bc3253460"
format_stamp: "Formatted at 2026-09-29 03:20:23 on dist-test-slave-c9qh"
server_key: "befe8cef71909dd162379b1998d3d44c"
server_key_iv: "611115b899673601adc691edbc0b0d40"
server_key_version: "encryptionkey@0"
I20260929 03:21:04.934080 5188 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:21:04.949345 5188 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:21:04.951788 5188 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:21:04.951933 5188 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:21:04.952176 5188 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:21:04.952682 5244 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260929 03:21:04.954710 5188 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260929 03:21:04.954787 5188 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260929 03:21:04.954834 5188 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260929 03:21:04.955720 5188 ts_tablet_manager.cc:610] Registered 1 tablets
I20260929 03:21:04.955777 5188 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260929 03:21:04.956229 5244 tablet_bootstrap.cc:492] T d00aff55fb054f128e16006aaeac20fc P f3c3608dbc5146a5affe080bc3253460: Bootstrap starting.
I20260929 03:21:04.962684 5188 rpc_server.cc:307] RPC server started. Bound to: 127.20.178.65:45779
I20260929 03:21:04.962759 5351 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.178.65:45779 every 8 connection(s)
I20260929 03:21:04.963219 5188 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/minicluster-data/ts-0/data/info.pb
I20260929 03:21:04.967163 5244 log.cc:824] T d00aff55fb054f128e16006aaeac20fc P f3c3608dbc5146a5affe080bc3253460: Log is configured to *not* fsync() on all Append() calls
I20260929 03:21:04.969370 21193 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuoaHbX/build/release/bin/kudu as pid 5188
I20260929 03:21:04.969522 5244 tablet_bootstrap.cc:492] T d00aff55fb054f128e16006aaeac20fc P f3c3608dbc5146a5affe080bc3253460: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 03:21:04.969689 21193 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuoaHbX/build/release/bin/kudu
/tmp/dist-test-taskuoaHbX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.178.66:43451
--local_ip_for_outbound_sockets=127.20.178.66
--tserver_master_addrs=127.20.178.126:36419
--webserver_port=34237
--webserver_interface=127.20.178.66
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.66.keytab
--principal=kudu/127.20.178.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.178.84:42735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 03:21:04.969892 5244 tablet_bootstrap.cc:492] T d00aff55fb054f128e16006aaeac20fc P f3c3608dbc5146a5affe080bc3253460: Bootstrap complete.
I20260929 03:21:04.970801 5244 ts_tablet_manager.cc:1397] T d00aff55fb054f128e16006aaeac20fc P f3c3608dbc5146a5affe080bc3253460: Time spent bootstrapping tablet: real 0.015s user 0.009s sys 0.003s
I20260929 03:21:04.972424 5244 raft_consensus.cc:348] T d00aff55fb054f128e16006aaeac20fc P f3c3608dbc5146a5affe080bc3253460 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15b1f41f721649dd8a97d314979ac521" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 33143 } } peers { permanent_uuid: "f3c3608dbc5146a5affe080bc3253460" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 45779 } } peers { permanent_uuid: "9978d1b39862456881b65ee097849482" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 43451 } }
I20260929 03:21:04.972702 5244 raft_consensus.cc:729] T d00aff55fb054f128e16006aaeac20fc P f3c3608dbc5146a5affe080bc3253460 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3c3608dbc5146a5affe080bc3253460, State: Initialized, Role: FOLLOWER
I20260929 03:21:04.972873 5244 consensus_queue.cc:260] T d00aff55fb054f128e16006aaeac20fc P f3c3608dbc5146a5affe080bc3253460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15b1f41f721649dd8a97d314979ac521" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 33143 } } peers { permanent_uuid: "f3c3608dbc5146a5affe080bc3253460" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 45779 } } peers { permanent_uuid: "9978d1b39862456881b65ee097849482" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 43451 } }
I20260929 03:21:04.973172 5244 ts_tablet_manager.cc:1428] T d00aff55fb054f128e16006aaeac20fc P f3c3608dbc5146a5affe080bc3253460: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
W20260929 03:21:05.054080 5354 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:21:04.964043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:36419 (local address 127.20.178.65:42359)
0929 03:21:04.964263 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0929 03:21:04.964313 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 03:21:04.964950 (+ 637us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:21:05.053860 (+ 88910us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.178.126:36419: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":106,"threads_started":1}
W20260929 03:21:05.054370 5352 heartbeater.cc:644] Failed to heartbeat to 127.20.178.126:36419 (0 consecutive failures): Network error: Failed to ping master at 127.20.178.126:36419: Client connection negotiation failed: client connection to 127.20.178.126:36419: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260929 03:21:05.054700 5354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:05.054577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:36419 (local address 127.20.178.65:44083)
0929 03:21:05.054632 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0929 03:21:05.054671 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.178.126:36419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260929 03:21:05.055091 5354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:05.054951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:36419 (local address 127.20.178.65:57585)
0929 03:21:05.055004 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0929 03:21:05.055065 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.178.126:36419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260929 03:21:05.055222 5352 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 03:21:05.066718 5356 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:21:05.066911 5356 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:21:05.066934 5356 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:21:05.068128 5356 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:21:05.068176 5356 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:21:05.068233 5356 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.178.66
I20260929 03:21:05.069761 5356 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.178.84:42735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.66.keytab
--principal=kudu/127.20.178.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.178.66:43451
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.178.66
--webserver_port=34237
--tserver_master_addrs=127.20.178.126:36419
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5356
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.178.66
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:21:05.069998 5356 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:21:05.070219 5356 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:21:05.075260 5362 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:21:05.075255 5363 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:21:05.075402 5365 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:21:05.075496 5356 server_base.cc:1034] running on GCE node
I20260929 03:21:05.075764 5356 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:21:05.076025 5356 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:21:05.077211 5356 hybrid_clock.cc:648] HybridClock initialized: now 1790652065077182 us; error 39 us; skew 500 ppm
Sep 29 03:21:05 dist-test-slave-c9qh krb5kdc[4050](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652065, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:21:05.079690 5356 init.cc:377] Logged in from keytab as kudu/127.20.178.66@KRBTEST.COM (short username kudu)
I20260929 03:21:05.080194 5356 webserver.cc:460] Webserver started at http://127.20.178.66:34237/ using document root <none> and password file <none>
I20260929 03:21:05.080435 5356 fs_manager.cc:359] Metadata directory not provided
I20260929 03:21:05.080505 5356 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:21:05.081943 5356 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260929 03:21:05.082643 5372 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:21:05.082827 5356 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260929 03:21:05.082919 5356 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/minicluster-data/ts-1/data,/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/minicluster-data/ts-1/wal
uuid: "9978d1b39862456881b65ee097849482"
format_stamp: "Formatted at 2026-09-29 03:20:23 on dist-test-slave-c9qh"
server_key: "aaefa21c358dc74638c24d23eda049bd"
server_key_iv: "82497572b79b0a0a6d8302733bc33410"
server_key_version: "encryptionkey@0"
I20260929 03:21:05.083303 5356 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:21:05.110163 5356 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:21:05.112691 5356 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:21:05.112859 5356 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:21:05.113122 5356 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:21:05.113467 5356 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:21:05.113520 5356 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:21:05.113562 5356 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:21:05.113605 5356 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:21:05.119576 5356 rpc_server.cc:307] RPC server started. Bound to: 127.20.178.66:43451
I20260929 03:21:05.119627 5485 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.178.66:43451 every 8 connection(s)
I20260929 03:21:05.119953 5356 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/minicluster-data/ts-1/data/info.pb
I20260929 03:21:05.120957 5488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:05.120590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:36419 (local address 127.20.178.66:34043)
0929 03:21:05.120742 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0929 03:21:05.120810 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.178.126:36419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":87,"threads_started":1}
W20260929 03:21:05.121376 5486 heartbeater.cc:644] Failed to heartbeat to 127.20.178.126:36419 (0 consecutive failures): Network error: Failed to ping master at 127.20.178.126:36419: Client connection negotiation failed: client connection to 127.20.178.126:36419: connect: Connection refused (error 111)
I20260929 03:21:05.121658 5488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:05.121573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:36419 (local address 127.20.178.66:54493)
0929 03:21:05.121616 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0929 03:21:05.121641 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.178.126:36419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260929 03:21:05.121920 5488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:05.121864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:36419 (local address 127.20.178.66:35835)
0929 03:21:05.121888 (+ 24us) negotiation.cc:107] Waiting for socket to connect
0929 03:21:05.121907 (+ 19us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.178.126:36419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":14}
W20260929 03:21:05.122035 5486 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260929 03:21:05.126117 21193 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuoaHbX/build/release/bin/kudu as pid 5356
I20260929 03:21:05.126406 21193 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuoaHbX/build/release/bin/kudu
/tmp/dist-test-taskuoaHbX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.178.67:33143
--local_ip_for_outbound_sockets=127.20.178.67
--tserver_master_addrs=127.20.178.126:36419
--webserver_port=45173
--webserver_interface=127.20.178.67
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.67.keytab
--principal=kudu/127.20.178.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.178.84:42735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:21:05.206954 5489 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:21:05.207159 5489 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:21:05.207185 5489 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:21:05.208367 5489 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:21:05.208408 5489 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:21:05.208434 5489 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.178.67
I20260929 03:21:05.209898 5489 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.178.84:42735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.67.keytab
--principal=kudu/127.20.178.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.178.67:33143
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.178.67
--webserver_port=45173
--tserver_master_addrs=127.20.178.126:36419
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5489
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.178.67
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:21:05.210130 5489 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:21:05.210371 5489 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:21:05.215497 5495 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:21:05.215551 5497 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:21:05.215551 5494 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:21:05.215605 5489 server_base.cc:1034] running on GCE node
I20260929 03:21:05.215946 5489 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:21:05.216193 5489 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:21:05.217334 5489 hybrid_clock.cc:648] HybridClock initialized: now 1790652065217312 us; error 39 us; skew 500 ppm
Sep 29 03:21:05 dist-test-slave-c9qh krb5kdc[4050](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652065, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:21:05.219792 5489 init.cc:377] Logged in from keytab as kudu/127.20.178.67@KRBTEST.COM (short username kudu)
I20260929 03:21:05.220264 5489 webserver.cc:460] Webserver started at http://127.20.178.67:45173/ using document root <none> and password file <none>
I20260929 03:21:05.220508 5489 fs_manager.cc:359] Metadata directory not provided
I20260929 03:21:05.220554 5489 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:21:05.221974 5489 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 03:21:05.222628 5504 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:21:05.222851 5489 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 03:21:05.222921 5489 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/minicluster-data/ts-2/data,/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/minicluster-data/ts-2/wal
uuid: "15b1f41f721649dd8a97d314979ac521"
format_stamp: "Formatted at 2026-09-29 03:20:23 on dist-test-slave-c9qh"
server_key: "dca81f8dcb62cb70aa55e2579b502690"
server_key_iv: "3565d2f4dcd71b8419b834ddb84a7a6d"
server_key_version: "encryptionkey@0"
I20260929 03:21:05.223299 5489 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:21:05.245371 5489 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:21:05.247478 5489 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:21:05.247622 5489 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:21:05.247884 5489 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:21:05.248200 5489 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:21:05.248258 5489 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:21:05.248317 5489 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:21:05.248338 5489 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:21:05.254114 5489 rpc_server.cc:307] RPC server started. Bound to: 127.20.178.67:33143
I20260929 03:21:05.254164 5617 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.178.67:33143 every 8 connection(s)
I20260929 03:21:05.254494 5489 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651852476208-21193-0/minicluster-data/ts-2/data/info.pb
I20260929 03:21:05.255541 5620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:05.255156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:36419 (local address 127.20.178.67:42721)
0929 03:21:05.255319 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0929 03:21:05.255394 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.178.126:36419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":82,"threads_started":1}
W20260929 03:21:05.255977 5618 heartbeater.cc:644] Failed to heartbeat to 127.20.178.126:36419 (0 consecutive failures): Network error: Failed to ping master at 127.20.178.126:36419: Client connection negotiation failed: client connection to 127.20.178.126:36419: connect: Connection refused (error 111)
I20260929 03:21:05.256232 5620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:05.256153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:36419 (local address 127.20.178.67:38497)
0929 03:21:05.256184 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0929 03:21:05.256213 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.178.126:36419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
I20260929 03:21:05.256518 5620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:05.256454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:36419 (local address 127.20.178.67:50971)
0929 03:21:05.256487 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0929 03:21:05.256506 (+ 19us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.178.126:36419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260929 03:21:05.256691 5618 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260929 03:21:05.261386 21193 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuoaHbX/build/release/bin/kudu as pid 5489
I20260929 03:21:06.055913 5622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:06.055664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:36419 (local address 127.20.178.65:51055)
0929 03:21:06.055813 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0929 03:21:06.055851 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.178.126:36419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":64,"threads_started":1}
I20260929 03:21:06.122601 5623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:06.122396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:36419 (local address 127.20.178.66:37959)
0929 03:21:06.122513 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0929 03:21:06.122549 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.178.126:36419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":50,"threads_started":1}
I20260929 03:21:06.257275 5624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:06.257047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:36419 (local address 127.20.178.67:48047)
0929 03:21:06.257187 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0929 03:21:06.257222 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.178.126:36419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":84,"threads_started":1}
I20260929 03:21:06.324777 5625 raft_consensus.cc:482] T d00aff55fb054f128e16006aaeac20fc P f3c3608dbc5146a5affe080bc3253460 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:21:06.324890 5625 raft_consensus.cc:504] T d00aff55fb054f128e16006aaeac20fc P f3c3608dbc5146a5affe080bc3253460 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15b1f41f721649dd8a97d314979ac521" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 33143 } } peers { permanent_uuid: "f3c3608dbc5146a5affe080bc3253460" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 45779 } } peers { permanent_uuid: "9978d1b39862456881b65ee097849482" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 43451 } }
I20260929 03:21:06.325261 5625 leader_election.cc:288] T d00aff55fb054f128e16006aaeac20fc P f3c3608dbc5146a5affe080bc3253460 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 15b1f41f721649dd8a97d314979ac521 (127.20.178.67:33143), 9978d1b39862456881b65ee097849482 (127.20.178.66:43451)
Sep 29 03:21:06 dist-test-slave-c9qh krb5kdc[4050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652064, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.65@KRBTEST.COM for kudu/127.20.178.67@KRBTEST.COM
Sep 29 03:21:06 dist-test-slave-c9qh krb5kdc[4050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652064, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.65@KRBTEST.COM for kudu/127.20.178.66@KRBTEST.COM
I20260929 03:21:06.336094 5622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:06.325457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.66:43451 (local address 127.20.178.65:34143)
0929 03:21:06.325567 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0929 03:21:06.325590 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 03:21:06.325654 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:21:06.326503 (+ 849us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:21:06.326506 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:21:06.326900 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:21:06.327089 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:21:06.327095 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:21:06.328202 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:21:06.328203 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:21:06.329061 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:21:06.329064 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:21:06.329785 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:21:06.329786 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:21:06.329844 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:21:06.330382 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:21:06.330391 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:21:06.332408 (+ 2017us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:21:06.334668 (+ 2260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:21:06.334669 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:21:06.334669 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:21:06.334849 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:21:06.335130 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:21:06.335130 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:21:06.335131 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:21:06.335178 (+ 47us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:21:06.335624 (+ 446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:21:06.335624 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:21:06.335656 (+ 32us) client_negotiation.cc:769] Sending connection context
0929 03:21:06.335759 (+ 103us) client_negotiation.cc:240] Negotiation successful
0929 03:21:06.335970 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":88,"mutex_wait_us":57}
I20260929 03:21:06.336100 5628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:06.325728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.65:40267 (local address 127.20.178.67:33143)
0929 03:21:06.325855 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0929 03:21:06.325863 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 03:21:06.325891 (+ 28us) server_negotiation.cc:407] Connection header received
0929 03:21:06.326302 (+ 411us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:21:06.326304 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:21:06.326339 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:21:06.326513 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:21:06.327240 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:06.328078 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:06.329152 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:06.329554 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:06.329683 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:21:06.332016 (+ 2333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:21:06.332021 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:21:06.332021 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:21:06.332034 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:21:06.334287 (+ 2253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:21:06.334739 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:21:06.334741 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:21:06.334742 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:21:06.334830 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:21:06.335083 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:21:06.335084 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:21:06.335084 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:21:06.335421 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:21:06.335506 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0929 03:21:06.335844 (+ 338us) server_negotiation.cc:299] Negotiation successful
0929 03:21:06.335983 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":56,"threads_started":1}
I20260929 03:21:06.336100 5627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:06.325550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.65:34143 (local address 127.20.178.66:43451)
0929 03:21:06.325766 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0929 03:21:06.325777 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 03:21:06.325804 (+ 27us) server_negotiation.cc:407] Connection header received
0929 03:21:06.326272 (+ 468us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:21:06.326274 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:21:06.326319 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:21:06.326513 (+ 194us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:21:06.327220 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:06.328084 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:06.329155 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:06.329553 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:06.329682 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:21:06.332523 (+ 2841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:21:06.332527 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:21:06.332527 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:21:06.332558 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:21:06.334577 (+ 2019us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:21:06.334945 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:21:06.334947 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:21:06.334947 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:21:06.335024 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:21:06.335275 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:21:06.335276 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:21:06.335276 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:21:06.335560 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:21:06.335617 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0929 03:21:06.335844 (+ 227us) server_negotiation.cc:299] Negotiation successful
0929 03:21:06.335983 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":51,"threads_started":1}
I20260929 03:21:06.336114 5626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:06.325486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.67:33143 (local address 127.20.178.65:40267)
0929 03:21:06.325611 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0929 03:21:06.325623 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 03:21:06.325681 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:21:06.326507 (+ 826us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:21:06.326508 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:21:06.326937 (+ 429us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:21:06.327089 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:21:06.327095 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:21:06.328215 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:21:06.328216 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:21:06.329060 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:21:06.329062 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:21:06.329653 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:21:06.329654 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:21:06.329764 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:21:06.330433 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:21:06.330438 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:21:06.331896 (+ 1458us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:21:06.334381 (+ 2485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:21:06.334388 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:21:06.334390 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:21:06.334647 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:21:06.334899 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:21:06.334900 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:21:06.334900 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:21:06.335014 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:21:06.335495 (+ 481us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:21:06.335501 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:21:06.335641 (+ 140us) client_negotiation.cc:769] Sending connection context
0929 03:21:06.335768 (+ 127us) client_negotiation.cc:240] Negotiation successful
0929 03:21:06.336024 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"mutex_wait_us":138,"thread_start_us":66,"threads_started":1}
W20260929 03:21:06.337723 5241 leader_election.cc:341] T d00aff55fb054f128e16006aaeac20fc P f3c3608dbc5146a5affe080bc3253460 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9978d1b39862456881b65ee097849482 (127.20.178.66:43451): Not found: Tablet not found: d00aff55fb054f128e16006aaeac20fc
W20260929 03:21:06.337813 5239 leader_election.cc:341] T d00aff55fb054f128e16006aaeac20fc P f3c3608dbc5146a5affe080bc3253460 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 15b1f41f721649dd8a97d314979ac521 (127.20.178.67:33143): Not found: Tablet not found: d00aff55fb054f128e16006aaeac20fc
I20260929 03:21:06.337882 5241 leader_election.cc:302] T d00aff55fb054f128e16006aaeac20fc P f3c3608dbc5146a5affe080bc3253460 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f3c3608dbc5146a5affe080bc3253460; no voters: 15b1f41f721649dd8a97d314979ac521, 9978d1b39862456881b65ee097849482
I20260929 03:21:06.338008 5625 raft_consensus.cc:2726] T d00aff55fb054f128e16006aaeac20fc P f3c3608dbc5146a5affe080bc3253460 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 03:21:07.056842 5629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:07.056492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:36419 (local address 127.20.178.65:58263)
0929 03:21:07.056745 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0929 03:21:07.056793 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.178.126:36419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":66,"threads_started":1}
I20260929 03:21:07.123342 5630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:07.123099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:36419 (local address 127.20.178.66:37247)
0929 03:21:07.123259 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0929 03:21:07.123298 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.178.126:36419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":89,"threads_started":1}
I20260929 03:21:07.258091 5631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:07.257774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:36419 (local address 127.20.178.67:58119)
0929 03:21:07.258003 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0929 03:21:07.258043 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.178.126:36419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":150,"threads_started":1}
I20260929 03:21:07.877594 5632 raft_consensus.cc:482] T d00aff55fb054f128e16006aaeac20fc P f3c3608dbc5146a5affe080bc3253460 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:21:07.877727 5632 raft_consensus.cc:504] T d00aff55fb054f128e16006aaeac20fc P f3c3608dbc5146a5affe080bc3253460 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15b1f41f721649dd8a97d314979ac521" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 33143 } } peers { permanent_uuid: "f3c3608dbc5146a5affe080bc3253460" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 45779 } } peers { permanent_uuid: "9978d1b39862456881b65ee097849482" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 43451 } }
I20260929 03:21:07.878007 5632 leader_election.cc:288] T d00aff55fb054f128e16006aaeac20fc P f3c3608dbc5146a5affe080bc3253460 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 15b1f41f721649dd8a97d314979ac521 (127.20.178.67:33143), 9978d1b39862456881b65ee097849482 (127.20.178.66:43451)
W20260929 03:21:07.878516 5241 leader_election.cc:341] T d00aff55fb054f128e16006aaeac20fc P f3c3608dbc5146a5affe080bc3253460 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9978d1b39862456881b65ee097849482 (127.20.178.66:43451): Not found: Tablet not found: d00aff55fb054f128e16006aaeac20fc
W20260929 03:21:07.878607 5239 leader_election.cc:341] T d00aff55fb054f128e16006aaeac20fc P f3c3608dbc5146a5affe080bc3253460 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 15b1f41f721649dd8a97d314979ac521 (127.20.178.67:33143): Not found: Tablet not found: d00aff55fb054f128e16006aaeac20fc
I20260929 03:21:07.878672 5239 leader_election.cc:302] T d00aff55fb054f128e16006aaeac20fc P f3c3608dbc5146a5affe080bc3253460 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f3c3608dbc5146a5affe080bc3253460; no voters: 15b1f41f721649dd8a97d314979ac521, 9978d1b39862456881b65ee097849482
I20260929 03:21:07.878762 5632 raft_consensus.cc:2726] T d00aff55fb054f128e16006aaeac20fc P f3c3608dbc5146a5affe080bc3253460 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 03:21:08.057667 5633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:08.057401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:36419 (local address 127.20.178.65:33875)
0929 03:21:08.057566 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0929 03:21:08.057609 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.178.126:36419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":76,"threads_started":1}
I20260929 03:21:08.123991 5634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:08.123798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:36419 (local address 127.20.178.66:46061)
0929 03:21:08.123912 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0929 03:21:08.123947 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.178.126:36419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":61,"threads_started":1}
I20260929 03:21:08.258903 5635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:08.258680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:36419 (local address 127.20.178.67:51997)
0929 03:21:08.258818 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0929 03:21:08.258858 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.178.126:36419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":64,"threads_started":1}
I20260929 03:21:09.058525 5636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:09.058263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:36419 (local address 127.20.178.65:47419)
0929 03:21:09.058428 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0929 03:21:09.058473 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.178.126:36419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":74,"threads_started":1}
I20260929 03:21:09.124778 5637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:09.124516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:36419 (local address 127.20.178.66:41835)
0929 03:21:09.124686 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0929 03:21:09.124736 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.178.126:36419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":58,"threads_started":1}
I20260929 03:21:09.259615 5638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:09.259374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:36419 (local address 127.20.178.67:57455)
0929 03:21:09.259535 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0929 03:21:09.259573 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.178.126:36419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":70,"threads_started":1}
I20260929 03:21:10.059366 5639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:10.059125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:36419 (local address 127.20.178.65:47397)
0929 03:21:10.059271 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0929 03:21:10.059317 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.178.126:36419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":64,"threads_started":1}
I20260929 03:21:10.125478 5640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:10.125255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:36419 (local address 127.20.178.66:50993)
0929 03:21:10.125391 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0929 03:21:10.125431 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.178.126:36419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":60,"threads_started":1}
I20260929 03:21:10.260290 5641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:10.260087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:36419 (local address 127.20.178.67:40223)
0929 03:21:10.260208 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0929 03:21:10.260247 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.178.126:36419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":57,"threads_started":1}
I20260929 03:21:11.060127 5642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:11.059911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:36419 (local address 127.20.178.65:46891)
0929 03:21:11.060040 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0929 03:21:11.060079 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.178.126:36419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":66,"threads_started":1}
I20260929 03:21:11.088313 5643 raft_consensus.cc:482] T d00aff55fb054f128e16006aaeac20fc P f3c3608dbc5146a5affe080bc3253460 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:21:11.088438 5643 raft_consensus.cc:504] T d00aff55fb054f128e16006aaeac20fc P f3c3608dbc5146a5affe080bc3253460 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15b1f41f721649dd8a97d314979ac521" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 33143 } } peers { permanent_uuid: "f3c3608dbc5146a5affe080bc3253460" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 45779 } } peers { permanent_uuid: "9978d1b39862456881b65ee097849482" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 43451 } }
I20260929 03:21:11.088692 5643 leader_election.cc:288] T d00aff55fb054f128e16006aaeac20fc P f3c3608dbc5146a5affe080bc3253460 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 15b1f41f721649dd8a97d314979ac521 (127.20.178.67:33143), 9978d1b39862456881b65ee097849482 (127.20.178.66:43451)
W20260929 03:21:11.089211 5241 leader_election.cc:341] T d00aff55fb054f128e16006aaeac20fc P f3c3608dbc5146a5affe080bc3253460 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9978d1b39862456881b65ee097849482 (127.20.178.66:43451): Not found: Tablet not found: d00aff55fb054f128e16006aaeac20fc
W20260929 03:21:11.089287 5239 leader_election.cc:341] T d00aff55fb054f128e16006aaeac20fc P f3c3608dbc5146a5affe080bc3253460 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 15b1f41f721649dd8a97d314979ac521 (127.20.178.67:33143): Not found: Tablet not found: d00aff55fb054f128e16006aaeac20fc
I20260929 03:21:11.089351 5239 leader_election.cc:302] T d00aff55fb054f128e16006aaeac20fc P f3c3608dbc5146a5affe080bc3253460 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f3c3608dbc5146a5affe080bc3253460; no voters: 15b1f41f721649dd8a97d314979ac521, 9978d1b39862456881b65ee097849482
I20260929 03:21:11.089449 5643 raft_consensus.cc:2726] T d00aff55fb054f128e16006aaeac20fc P f3c3608dbc5146a5affe080bc3253460 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 03:21:11.126214 5644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:11.125964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:36419 (local address 127.20.178.66:36569)
0929 03:21:11.126079 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0929 03:21:11.126142 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.178.126:36419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":55,"threads_started":1}
I20260929 03:21:11.261138 5645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:11.260887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:36419 (local address 127.20.178.67:48283)
0929 03:21:11.261054 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0929 03:21:11.261096 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.178.126:36419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":89,"threads_started":1}
I20260929 03:21:12.060961 5646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:12.060703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:36419 (local address 127.20.178.65:54275)
0929 03:21:12.060868 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0929 03:21:12.060912 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.178.126:36419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":77,"threads_started":1}
I20260929 03:21:12.126823 5647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:12.126602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:36419 (local address 127.20.178.66:48299)
0929 03:21:12.126746 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0929 03:21:12.126783 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.178.126:36419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":55,"threads_started":1}
I20260929 03:21:12.261847 5648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:12.261641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:36419 (local address 127.20.178.67:57497)
0929 03:21:12.261764 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0929 03:21:12.261802 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.178.126:36419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":61,"threads_started":1}
I20260929 03:21:13.061671 5649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:13.061451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:36419 (local address 127.20.178.65:58223)
0929 03:21:13.061585 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0929 03:21:13.061622 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.178.126:36419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":60,"threads_started":1}
I20260929 03:21:13.127529 5650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:13.127279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:36419 (local address 127.20.178.66:43697)
0929 03:21:13.127435 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0929 03:21:13.127485 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.178.126:36419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":80,"threads_started":1}
I20260929 03:21:13.262519 5651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:13.262319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:36419 (local address 127.20.178.67:47315)
0929 03:21:13.262442 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0929 03:21:13.262477 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.178.126:36419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":60,"threads_started":1}
I20260929 03:21:14.062373 5652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:14.062133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:36419 (local address 127.20.178.65:59191)
0929 03:21:14.062291 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0929 03:21:14.062325 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.178.126:36419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":62,"threads_started":1}
I20260929 03:21:14.128208 5653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:14.127949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:36419 (local address 127.20.178.66:49105)
0929 03:21:14.128090 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0929 03:21:14.128127 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.178.126:36419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":58,"threads_started":1}
I20260929 03:21:14.263177 5654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:14.262974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:36419 (local address 127.20.178.67:37789)
0929 03:21:14.263095 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0929 03:21:14.263131 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.178.126:36419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":61,"threads_started":1}
I20260929 03:21:15.063184 5655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:15.062929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:36419 (local address 127.20.178.65:38951)
0929 03:21:15.063079 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0929 03:21:15.063132 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.178.126:36419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":66,"threads_started":1}
I20260929 03:21:15.128988 5656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:15.128749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:36419 (local address 127.20.178.66:42095)
0929 03:21:15.128903 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0929 03:21:15.128941 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.178.126:36419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":73,"threads_started":1}
I20260929 03:21:15.263881 5657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:15.263656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:36419 (local address 127.20.178.67:49453)
0929 03:21:15.263796 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0929 03:21:15.263835 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.178.126:36419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":80,"threads_started":1}
I20260929 03:21:15.298830 5658 raft_consensus.cc:482] T d00aff55fb054f128e16006aaeac20fc P f3c3608dbc5146a5affe080bc3253460 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:21:15.298954 5658 raft_consensus.cc:504] T d00aff55fb054f128e16006aaeac20fc P f3c3608dbc5146a5affe080bc3253460 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15b1f41f721649dd8a97d314979ac521" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 33143 } } peers { permanent_uuid: "f3c3608dbc5146a5affe080bc3253460" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 45779 } } peers { permanent_uuid: "9978d1b39862456881b65ee097849482" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 43451 } }
I20260929 03:21:15.299163 5658 leader_election.cc:288] T d00aff55fb054f128e16006aaeac20fc P f3c3608dbc5146a5affe080bc3253460 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 15b1f41f721649dd8a97d314979ac521 (127.20.178.67:33143), 9978d1b39862456881b65ee097849482 (127.20.178.66:43451)
W20260929 03:21:15.299530 5241 leader_election.cc:341] T d00aff55fb054f128e16006aaeac20fc P f3c3608dbc5146a5affe080bc3253460 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9978d1b39862456881b65ee097849482 (127.20.178.66:43451): Not found: Tablet not found: d00aff55fb054f128e16006aaeac20fc
W20260929 03:21:15.299602 5239 leader_election.cc:341] T d00aff55fb054f128e16006aaeac20fc P f3c3608dbc5146a5affe080bc3253460 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 15b1f41f721649dd8a97d314979ac521 (127.20.178.67:33143): Not found: Tablet not found: d00aff55fb054f128e16006aaeac20fc
I20260929 03:21:15.299702 5239 leader_election.cc:302] T d00aff55fb054f128e16006aaeac20fc P f3c3608dbc5146a5affe080bc3253460 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f3c3608dbc5146a5affe080bc3253460; no voters: 15b1f41f721649dd8a97d314979ac521, 9978d1b39862456881b65ee097849482
I20260929 03:21:15.299834 5658 raft_consensus.cc:2726] T d00aff55fb054f128e16006aaeac20fc P f3c3608dbc5146a5affe080bc3253460 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 03:21:16.063934 5659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:16.063710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:36419 (local address 127.20.178.65:49909)
0929 03:21:16.063853 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0929 03:21:16.063888 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.178.126:36419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":63,"threads_started":1}
I20260929 03:21:16.129698 5660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:16.129493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:36419 (local address 127.20.178.66:58275)
0929 03:21:16.129612 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0929 03:21:16.129650 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.178.126:36419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":57,"threads_started":1}
I20260929 03:21:16.264564 5661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:16.264343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:36419 (local address 127.20.178.67:44471)
0929 03:21:16.264483 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0929 03:21:16.264519 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.178.126:36419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":72,"threads_started":1}
I20260929 03:21:17.064790 5662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:17.064486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:36419 (local address 127.20.178.65:41315)
0929 03:21:17.064692 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0929 03:21:17.064740 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.178.126:36419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":69,"threads_started":1}
I20260929 03:21:17.130388 5663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:17.130161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:36419 (local address 127.20.178.66:38297)
0929 03:21:17.130303 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0929 03:21:17.130343 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.178.126:36419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":58,"threads_started":1}
I20260929 03:21:17.265304 5664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:17.265048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:36419 (local address 127.20.178.67:52627)
0929 03:21:17.265210 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0929 03:21:17.265255 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.178.126:36419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":65,"threads_started":1}
I20260929 03:21:18.065675 5665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:18.065422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:36419 (local address 127.20.178.65:47061)
0929 03:21:18.065585 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0929 03:21:18.065627 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.178.126:36419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":83,"threads_started":1}
I20260929 03:21:18.131062 5666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:18.130840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:36419 (local address 127.20.178.66:59011)
0929 03:21:18.130977 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0929 03:21:18.131018 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.178.126:36419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":57,"threads_started":1}
I20260929 03:21:18.266096 5667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:18.265841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:36419 (local address 127.20.178.67:39651)
0929 03:21:18.266005 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0929 03:21:18.266047 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.178.126:36419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":78,"threads_started":1}
I20260929 03:21:19.066561 5668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:19.066326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:36419 (local address 127.20.178.65:58491)
0929 03:21:19.066458 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0929 03:21:19.066513 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.178.126:36419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":64,"threads_started":1}
I20260929 03:21:19.131757 5669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:19.131545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:36419 (local address 127.20.178.66:41619)
0929 03:21:19.131665 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0929 03:21:19.131712 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.178.126:36419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":56,"threads_started":1}
I20260929 03:21:19.266881 5670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:19.266647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:36419 (local address 127.20.178.67:33593)
0929 03:21:19.266793 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0929 03:21:19.266834 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.178.126:36419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":64,"threads_started":1}
I20260929 03:21:20.067415 5671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:20.067184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:36419 (local address 127.20.178.65:35469)
0929 03:21:20.067337 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0929 03:21:20.067371 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.178.126:36419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":95,"threads_started":1}
I20260929 03:21:20.132580 5672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:20.132333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:36419 (local address 127.20.178.66:36969)
0929 03:21:20.132471 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0929 03:21:20.132512 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.178.126:36419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":85,"threads_started":1}
I20260929 03:21:20.263620 21193 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuoaHbX/build/release/bin/kudu with pid 5188
I20260929 03:21:20.267771 5673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:20.267528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:36419 (local address 127.20.178.67:59231)
0929 03:21:20.267657 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0929 03:21:20.267704 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.178.126:36419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":57,"threads_started":1}
I20260929 03:21:20.268755 21193 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuoaHbX/build/release/bin/kudu with pid 5356
I20260929 03:21:20.273025 21193 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuoaHbX/build/release/bin/kudu with pid 5489
2026-09-29T03:21:20Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (58133 ms)
[----------] 30 tests from SecurityITest (227803 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651852476208-21193-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:21:20 dist-test-slave-c9qh krb5kdc[5676](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:21:20 dist-test-slave-c9qh krb5kdc[5676](info): set up 2 sockets
Sep 29 03:21:20 dist-test-slave-c9qh krb5kdc[5676](info): commencing operation
krb5kdc: starting...
W20260929 03:21:21.308385 21193 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.001s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651852476208-21193-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651852476208-21193-0/krb5kdc/test-user.keytab.
Sep 29 03:21:21 dist-test-slave-c9qh krb5kdc[5676](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652081, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:21:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:21:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.178.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.178.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.178.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.126.keytab.
Entry for principal kudu/127.20.178.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.126.keytab.
I20260929 03:21:21.425257 21193 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuoaHbX/build/release/bin/kudu
/tmp/dist-test-taskuoaHbX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651852476208-21193-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651852476208-21193-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651852476208-21193-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651852476208-21193-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.178.126:41855
--webserver_interface=127.20.178.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.126.keytab
--principal=kudu/127.20.178.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.178.84:44761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.178.126:41855
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651852476208-21193-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651852476208-21193-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651852476208-21193-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:21:21.505332 5689 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:21:21.505550 5689 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:21:21.505573 5689 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:21:21.506745 5689 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:21:21.506788 5689 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:21:21.506803 5689 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:21:21.506845 5689 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:21:21.506875 5689 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:21:21.508538 5689 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.178.84:44761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651852476208-21193-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651852476208-21193-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.178.126:41855
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.126.keytab
--principal=kudu/127.20.178.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.178.126:41855
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651852476208-21193-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.178.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5689
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651852476208-21193-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:21:21.508828 5689 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:21:21.509127 5689 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:21:21.514191 5694 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:21:21.514219 5697 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:21:21.514264 5695 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:21:21.514284 5689 server_base.cc:1034] running on GCE node
I20260929 03:21:21.514642 5689 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:21:21.514950 5689 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:21:21.516117 5689 hybrid_clock.cc:648] HybridClock initialized: now 1790652081516098 us; error 40 us; skew 500 ppm
Sep 29 03:21:21 dist-test-slave-c9qh krb5kdc[5676](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652081, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:21:21.518679 5689 init.cc:377] Logged in from keytab as kudu/127.20.178.126@KRBTEST.COM (short username kudu)
I20260929 03:21:21.519126 5689 webserver.cc:460] Webserver started at http://127.20.178.126:39589/ using document root <none> and password file <none>
I20260929 03:21:21.519402 5689 fs_manager.cc:359] Metadata directory not provided
I20260929 03:21:21.519449 5689 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:21:21.519593 5689 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:21:21.520697 5689 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651852476208-21193-0/minicluster-data/master-0/data/instance:
uuid: "2c67e8a370c547198280b321887ff8cf"
format_stamp: "Formatted at 2026-09-29 03:21:21 on dist-test-slave-c9qh"
server_key: "ee1b51a4a0cdc0e6e8bc8c25d47aeb27"
server_key_iv: "0071891cc2594fb79be45aae25c6ead1"
server_key_version: "encryptionkey@0"
I20260929 03:21:21.521072 5689 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651852476208-21193-0/minicluster-data/master-0/wal/instance:
uuid: "2c67e8a370c547198280b321887ff8cf"
format_stamp: "Formatted at 2026-09-29 03:21:21 on dist-test-slave-c9qh"
server_key: "ee1b51a4a0cdc0e6e8bc8c25d47aeb27"
server_key_iv: "0071891cc2594fb79be45aae25c6ead1"
server_key_version: "encryptionkey@0"
I20260929 03:21:21.522353 5689 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 03:21:21.523108 5704 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:21:21.523303 5689 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 03:21:21.523391 5689 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651852476208-21193-0/minicluster-data/master-0/data,/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651852476208-21193-0/minicluster-data/master-0/wal
uuid: "2c67e8a370c547198280b321887ff8cf"
format_stamp: "Formatted at 2026-09-29 03:21:21 on dist-test-slave-c9qh"
server_key: "ee1b51a4a0cdc0e6e8bc8c25d47aeb27"
server_key_iv: "0071891cc2594fb79be45aae25c6ead1"
server_key_version: "encryptionkey@0"
I20260929 03:21:21.523443 5689 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651852476208-21193-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651852476208-21193-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651852476208-21193-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:21:21.549990 5689 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:21:21.551918 5689 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:21:21.552027 5689 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:21:21.555817 5689 rpc_server.cc:307] RPC server started. Bound to: 127.20.178.126:41855
I20260929 03:21:21.555884 5756 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.178.126:41855 every 8 connection(s)
I20260929 03:21:21.556362 5689 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651852476208-21193-0/minicluster-data/master-0/data/info.pb
I20260929 03:21:21.557135 5757 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:21:21.559499 5757 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c67e8a370c547198280b321887ff8cf: Bootstrap starting.
I20260929 03:21:21.559845 21193 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuoaHbX/build/release/bin/kudu as pid 5689
I20260929 03:21:21.559930 21193 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651852476208-21193-0/minicluster-data/master-0/wal/instance
I20260929 03:21:21.560109 21193 external_mini_cluster.cc:1428] Setting key c4317b8e8ae7eaccc296a60ffe50c10d
I20260929 03:21:21.560118 5757 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2c67e8a370c547198280b321887ff8cf: Neither blocks nor log segments found. Creating new log.
I20260929 03:21:21.560511 5757 log.cc:824] T 00000000000000000000000000000000 P 2c67e8a370c547198280b321887ff8cf: Log is configured to *not* fsync() on all Append() calls
I20260929 03:21:21.561271 5757 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c67e8a370c547198280b321887ff8cf: No bootstrap required, opened a new log
I20260929 03:21:21.563108 5757 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2c67e8a370c547198280b321887ff8cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c67e8a370c547198280b321887ff8cf" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 41855 } }
I20260929 03:21:21.563269 5757 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2c67e8a370c547198280b321887ff8cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:21:21.563315 5757 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2c67e8a370c547198280b321887ff8cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c67e8a370c547198280b321887ff8cf, State: Initialized, Role: FOLLOWER
I20260929 03:21:21.563485 5757 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2c67e8a370c547198280b321887ff8cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c67e8a370c547198280b321887ff8cf" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 41855 } }
I20260929 03:21:21.563580 5757 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2c67e8a370c547198280b321887ff8cf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:21:21.563624 5757 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2c67e8a370c547198280b321887ff8cf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:21:21.563671 5757 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2c67e8a370c547198280b321887ff8cf [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:21:21.564191 5757 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2c67e8a370c547198280b321887ff8cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c67e8a370c547198280b321887ff8cf" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 41855 } }
I20260929 03:21:21.564307 5757 leader_election.cc:302] T 00000000000000000000000000000000 P 2c67e8a370c547198280b321887ff8cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2c67e8a370c547198280b321887ff8cf; no voters:
I20260929 03:21:21.564492 5757 leader_election.cc:288] T 00000000000000000000000000000000 P 2c67e8a370c547198280b321887ff8cf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:21:21.564611 5762 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2c67e8a370c547198280b321887ff8cf [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:21:21.564797 5757 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2c67e8a370c547198280b321887ff8cf [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:21:21.564847 5762 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2c67e8a370c547198280b321887ff8cf [term 1 LEADER]: Becoming Leader. State: Replica: 2c67e8a370c547198280b321887ff8cf, State: Running, Role: LEADER
I20260929 03:21:21.564983 5762 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2c67e8a370c547198280b321887ff8cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c67e8a370c547198280b321887ff8cf" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 41855 } }
I20260929 03:21:21.565388 5763 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c67e8a370c547198280b321887ff8cf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2c67e8a370c547198280b321887ff8cf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c67e8a370c547198280b321887ff8cf" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 41855 } } }
I20260929 03:21:21.565464 5763 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c67e8a370c547198280b321887ff8cf [sys.catalog]: This master's current role is: LEADER
I20260929 03:21:21.565726 5764 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c67e8a370c547198280b321887ff8cf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2c67e8a370c547198280b321887ff8cf. Latest consensus state: current_term: 1 leader_uuid: "2c67e8a370c547198280b321887ff8cf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c67e8a370c547198280b321887ff8cf" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 41855 } } }
I20260929 03:21:21.565738 5771 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:21:21.566316 5771 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:21:21.565816 5764 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c67e8a370c547198280b321887ff8cf [sys.catalog]: This master's current role is: LEADER
Sep 29 03:21:21 dist-test-slave-c9qh krb5kdc[5676](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652081, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.178.126@KRBTEST.COM
I20260929 03:21:21.568696 5771 catalog_manager.cc:1347] Generated new cluster ID: 28deb385f28b4b8ab1ba32341b084d8b
I20260929 03:21:21.568746 5771 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:21:21.572034 5761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:21.560669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55884 (local address 127.20.178.126:41855)
0929 03:21:21.560884 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0929 03:21:21.560895 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 03:21:21.560928 (+ 33us) server_negotiation.cc:407] Connection header received
0929 03:21:21.561441 (+ 513us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:21:21.561451 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:21:21.561516 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:21:21.561675 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:21:21.562336 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:21.563385 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:21.564363 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:21.565003 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:21.565452 (+ 449us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:21:21.568051 (+ 2599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:21:21.568054 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:21:21.568055 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:21:21.568064 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:21:21.570153 (+ 2089us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:21:21.570573 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:21:21.570575 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:21:21.570576 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:21:21.570677 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:21:21.570889 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:21:21.570890 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:21:21.570890 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:21:21.571155 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:21:21.571298 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0929 03:21:21.571419 (+ 121us) server_negotiation.cc:299] Negotiation successful
0929 03:21:21.571630 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":113,"threads_started":1}
I20260929 03:21:21.605146 5771 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:21:21.605795 5771 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:21:21.614022 5771 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2c67e8a370c547198280b321887ff8cf: Generated new TSK 0
I20260929 03:21:21.614226 5771 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.178.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.178.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.178.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.65.keytab.
Entry for principal kudu/127.20.178.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.65.keytab.
I20260929 03:21:21.644140 21193 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuoaHbX/build/release/bin/kudu
/tmp/dist-test-taskuoaHbX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651852476208-21193-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651852476208-21193-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651852476208-21193-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651852476208-21193-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.178.65:0
--local_ip_for_outbound_sockets=127.20.178.65
--webserver_interface=127.20.178.65
--webserver_port=0
--tserver_master_addrs=127.20.178.126:41855
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.65.keytab
--principal=kudu/127.20.178.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.178.84:44761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651852476208-21193-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651852476208-21193-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651852476208-21193-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:21:21.724378 5783 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:21:21.724576 5783 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:21:21.724601 5783 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:21:21.725775 5783 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:21:21.725814 5783 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:21:21.725841 5783 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.178.65
I20260929 03:21:21.727327 5783 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.178.84:44761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651852476208-21193-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651852476208-21193-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.65.keytab
--principal=kudu/127.20.178.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.178.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651852476208-21193-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.178.65
--webserver_port=0
--tserver_master_addrs=127.20.178.126:41855
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5783
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.178.65
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651852476208-21193-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:21:21.727555 5783 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:21:21.727814 5783 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:21:21.732873 5791 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:21:21.732843 5788 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:21:21.732913 5783 server_base.cc:1034] running on GCE node
W20260929 03:21:21.732854 5789 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:21:21.733242 5783 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:21:21.733500 5783 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:21:21.734668 5783 hybrid_clock.cc:648] HybridClock initialized: now 1790652081734653 us; error 46 us; skew 500 ppm
Sep 29 03:21:21 dist-test-slave-c9qh krb5kdc[5676](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652081, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:21:21.737164 5783 init.cc:377] Logged in from keytab as kudu/127.20.178.65@KRBTEST.COM (short username kudu)
I20260929 03:21:21.737581 5783 webserver.cc:460] Webserver started at http://127.20.178.65:35715/ using document root <none> and password file <none>
I20260929 03:21:21.737813 5783 fs_manager.cc:359] Metadata directory not provided
I20260929 03:21:21.737891 5783 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:21:21.738044 5783 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:21:21.739084 5783 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651852476208-21193-0/minicluster-data/ts-0/data/instance:
uuid: "42cfed0d018c4ec58518a3d749e62c70"
format_stamp: "Formatted at 2026-09-29 03:21:21 on dist-test-slave-c9qh"
server_key: "52f62516bcec0d9bb90743775f5f1b74"
server_key_iv: "53554de14f7ec94f423fafcd86377f9d"
server_key_version: "encryptionkey@0"
I20260929 03:21:21.739437 5783 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651852476208-21193-0/minicluster-data/ts-0/wal/instance:
uuid: "42cfed0d018c4ec58518a3d749e62c70"
format_stamp: "Formatted at 2026-09-29 03:21:21 on dist-test-slave-c9qh"
server_key: "52f62516bcec0d9bb90743775f5f1b74"
server_key_iv: "53554de14f7ec94f423fafcd86377f9d"
server_key_version: "encryptionkey@0"
I20260929 03:21:21.740756 5783 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260929 03:21:21.741536 5798 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:21:21.741707 5783 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 03:21:21.741804 5783 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651852476208-21193-0/minicluster-data/ts-0/data,/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651852476208-21193-0/minicluster-data/ts-0/wal
uuid: "42cfed0d018c4ec58518a3d749e62c70"
format_stamp: "Formatted at 2026-09-29 03:21:21 on dist-test-slave-c9qh"
server_key: "52f62516bcec0d9bb90743775f5f1b74"
server_key_iv: "53554de14f7ec94f423fafcd86377f9d"
server_key_version: "encryptionkey@0"
I20260929 03:21:21.741904 5783 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651852476208-21193-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651852476208-21193-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651852476208-21193-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:21:21.781522 5783 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:21:21.784472 5783 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:21:21.784648 5783 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:21:21.784907 5783 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:21:21.785243 5783 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:21:21.785305 5783 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:21:21.785341 5783 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:21:21.785364 5783 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:21:21.793025 5783 rpc_server.cc:307] RPC server started. Bound to: 127.20.178.65:43363
I20260929 03:21:21.793164 5911 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.178.65:43363 every 8 connection(s)
I20260929 03:21:21.793481 5783 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651852476208-21193-0/minicluster-data/ts-0/data/info.pb
I20260929 03:21:21.799331 21193 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuoaHbX/build/release/bin/kudu as pid 5783
I20260929 03:21:21.799443 21193 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651852476208-21193-0/minicluster-data/ts-0/wal/instance
Sep 29 03:21:21 dist-test-slave-c9qh krb5kdc[5676](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652081, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.65@KRBTEST.COM for kudu/127.20.178.126@KRBTEST.COM
I20260929 03:21:21.799638 21193 external_mini_cluster.cc:1428] Setting key 78dc0f3c96c627b1932d695d7575315e
I20260929 03:21:21.803227 5761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:21.794228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.65:49371 (local address 127.20.178.126:41855)
0929 03:21:21.794273 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0929 03:21:21.794275 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:21:21.794490 (+ 215us) server_negotiation.cc:407] Connection header received
0929 03:21:21.794960 (+ 470us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:21:21.794961 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:21:21.794971 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:21:21.795041 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:21:21.795957 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:21.796501 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:21.797543 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:21.797752 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:21.797810 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:21:21.800071 (+ 2261us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:21:21.800073 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:21:21.800073 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:21:21.800081 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:21:21.801866 (+ 1785us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:21:21.802246 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:21:21.802248 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:21:21.802248 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:21:21.802314 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:21:21.802557 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:21:21.802558 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:21:21.802558 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:21:21.802699 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:21:21.802776 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0929 03:21:21.803056 (+ 280us) server_negotiation.cc:299] Negotiation successful
0929 03:21:21.803136 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260929 03:21:21.803443 5914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:21.794192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:41855 (local address 127.20.178.65:49371)
0929 03:21:21.794356 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0929 03:21:21.794399 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 03:21:21.794876 (+ 477us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:21:21.795098 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:21:21.795101 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:21:21.795665 (+ 564us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:21:21.795862 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:21:21.795866 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:21:21.796587 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:21:21.796587 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:21:21.797466 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:21:21.797469 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:21:21.797828 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:21:21.797829 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:21:21.797932 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:21:21.798384 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:21:21.798392 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:21:21.799966 (+ 1574us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:21:21.801961 (+ 1995us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:21:21.801968 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:21:21.801969 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:21:21.802176 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:21:21.802383 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:21:21.802384 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:21:21.802384 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:21:21.802480 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:21:21.802782 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:21:21.802789 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:21:21.802926 (+ 137us) client_negotiation.cc:769] Sending connection context
0929 03:21:21.803035 (+ 109us) client_negotiation.cc:240] Negotiation successful
0929 03:21:21.803203 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":88,"threads_started":1}
I20260929 03:21:21.803907 5912 heartbeater.cc:347] Connected to a master server at 127.20.178.126:41855
I20260929 03:21:21.804001 5912 heartbeater.cc:464] Registering TS with master...
I20260929 03:21:21.804139 5912 heartbeater.cc:511] Master 127.20.178.126:41855 requested a full tablet report, sending...
I20260929 03:21:21.804733 5721 ts_manager.cc:194] Registered new tserver with Master: 42cfed0d018c4ec58518a3d749e62c70 (127.20.178.65:43363)
I20260929 03:21:21.805725 5721 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.178.65@KRBTEST.COM'} at 127.20.178.65:49371
WARNING: no policy specified for kudu/127.20.178.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.178.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.178.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.66.keytab.
Entry for principal kudu/127.20.178.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.66.keytab.
I20260929 03:21:21.825481 21193 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuoaHbX/build/release/bin/kudu
/tmp/dist-test-taskuoaHbX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651852476208-21193-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651852476208-21193-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651852476208-21193-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651852476208-21193-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.178.66:0
--local_ip_for_outbound_sockets=127.20.178.66
--webserver_interface=127.20.178.66
--webserver_port=0
--tserver_master_addrs=127.20.178.126:41855
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.66.keytab
--principal=kudu/127.20.178.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.178.84:44761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651852476208-21193-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651852476208-21193-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651852476208-21193-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:21:21.905175 5917 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:21:21.905367 5917 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:21:21.905390 5917 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:21:21.906454 5917 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:21:21.906487 5917 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:21:21.906512 5917 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.178.66
I20260929 03:21:21.907860 5917 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.178.84:44761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651852476208-21193-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651852476208-21193-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.66.keytab
--principal=kudu/127.20.178.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.178.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651852476208-21193-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.178.66
--webserver_port=0
--tserver_master_addrs=127.20.178.126:41855
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5917
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.178.66
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651852476208-21193-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:21:21.908041 5917 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:21:21.908201 5917 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:21:21.913074 5923 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:21:21.913049 5922 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:21:21.913213 5917 server_base.cc:1034] running on GCE node
W20260929 03:21:21.913082 5925 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:21:21.913511 5917 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:21:21.913769 5917 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:21:21.914927 5917 hybrid_clock.cc:648] HybridClock initialized: now 1790652081914916 us; error 42 us; skew 500 ppm
Sep 29 03:21:21 dist-test-slave-c9qh krb5kdc[5676](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652081, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:21:21.917275 5917 init.cc:377] Logged in from keytab as kudu/127.20.178.66@KRBTEST.COM (short username kudu)
I20260929 03:21:21.917729 5917 webserver.cc:460] Webserver started at http://127.20.178.66:42393/ using document root <none> and password file <none>
I20260929 03:21:21.917948 5917 fs_manager.cc:359] Metadata directory not provided
I20260929 03:21:21.918015 5917 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:21:21.918164 5917 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:21:21.919215 5917 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651852476208-21193-0/minicluster-data/ts-1/data/instance:
uuid: "2dc255534e484c37b63578b25bcae523"
format_stamp: "Formatted at 2026-09-29 03:21:21 on dist-test-slave-c9qh"
server_key: "6d29a31d716b99866f2e996cef1ee07d"
server_key_iv: "46c223baae99fa50ec116c4393e7726e"
server_key_version: "encryptionkey@0"
I20260929 03:21:21.919589 5917 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651852476208-21193-0/minicluster-data/ts-1/wal/instance:
uuid: "2dc255534e484c37b63578b25bcae523"
format_stamp: "Formatted at 2026-09-29 03:21:21 on dist-test-slave-c9qh"
server_key: "6d29a31d716b99866f2e996cef1ee07d"
server_key_iv: "46c223baae99fa50ec116c4393e7726e"
server_key_version: "encryptionkey@0"
I20260929 03:21:21.920933 5917 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 03:21:21.921646 5932 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:21:21.921808 5917 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 03:21:21.921914 5917 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651852476208-21193-0/minicluster-data/ts-1/data,/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651852476208-21193-0/minicluster-data/ts-1/wal
uuid: "2dc255534e484c37b63578b25bcae523"
format_stamp: "Formatted at 2026-09-29 03:21:21 on dist-test-slave-c9qh"
server_key: "6d29a31d716b99866f2e996cef1ee07d"
server_key_iv: "46c223baae99fa50ec116c4393e7726e"
server_key_version: "encryptionkey@0"
I20260929 03:21:21.921988 5917 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651852476208-21193-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651852476208-21193-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651852476208-21193-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:21:21.930295 5917 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:21:21.932056 5917 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:21:21.932211 5917 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:21:21.932425 5917 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:21:21.932780 5917 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:21:21.932837 5917 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:21:21.932878 5917 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:21:21.932904 5917 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:21:21.938930 5917 rpc_server.cc:307] RPC server started. Bound to: 127.20.178.66:41135
I20260929 03:21:21.938966 6045 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.178.66:41135 every 8 connection(s)
I20260929 03:21:21.939286 5917 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651852476208-21193-0/minicluster-data/ts-1/data/info.pb
I20260929 03:21:21.940119 21193 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuoaHbX/build/release/bin/kudu as pid 5917
I20260929 03:21:21.940233 21193 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651852476208-21193-0/minicluster-data/ts-1/wal/instance
I20260929 03:21:21.940457 21193 external_mini_cluster.cc:1428] Setting key 470389375b41b3ac4504b346c534ca57
Sep 29 03:21:21 dist-test-slave-c9qh krb5kdc[5676](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652081, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.66@KRBTEST.COM for kudu/127.20.178.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.20.178.67@KRBTEST.COM; defaulting to no policy
I20260929 03:21:21.949896 5761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:21.940067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.66:37549 (local address 127.20.178.126:41855)
0929 03:21:21.940141 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0929 03:21:21.940143 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:21:21.940400 (+ 257us) server_negotiation.cc:407] Connection header received
0929 03:21:21.941070 (+ 670us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:21:21.941072 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:21:21.941083 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:21:21.941149 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:21:21.942446 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:21.943135 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:21.944101 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:21.944447 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:21.944523 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:21:21.946782 (+ 2259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:21:21.946783 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:21:21.946783 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:21:21.946791 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:21:21.948412 (+ 1621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:21:21.948958 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:21:21.948959 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:21:21.948959 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:21:21.949026 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:21:21.949271 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:21:21.949272 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:21:21.949272 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:21:21.949441 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:21:21.949506 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0929 03:21:21.949761 (+ 255us) server_negotiation.cc:299] Negotiation successful
0929 03:21:21.949830 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260929 03:21:21.950043 6048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:21.939973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:41855 (local address 127.20.178.66:37549)
0929 03:21:21.940258 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0929 03:21:21.940306 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0929 03:21:21.940959 (+ 653us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:21:21.941231 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:21:21.941236 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:21:21.942064 (+ 828us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:21:21.942350 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:21:21.942356 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:21:21.943237 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:21:21.943237 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:21:21.944014 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:21:21.944015 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:21:21.944592 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:21:21.944594 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:21:21.944749 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:21:21.945342 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:21:21.945350 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:21:21.946687 (+ 1337us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:21:21.948566 (+ 1879us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:21:21.948572 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:21:21.948577 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:21:21.948881 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:21:21.949095 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:21:21.949095 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:21:21.949096 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:21:21.949196 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:21:21.949514 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:21:21.949520 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:21:21.949631 (+ 111us) client_negotiation.cc:769] Sending connection context
0929 03:21:21.949705 (+ 74us) client_negotiation.cc:240] Negotiation successful
0929 03:21:21.949826 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":106,"threads_started":1}
I20260929 03:21:21.950654 6046 heartbeater.cc:347] Connected to a master server at 127.20.178.126:41855
I20260929 03:21:21.950737 6046 heartbeater.cc:464] Registering TS with master...
I20260929 03:21:21.950884 6046 heartbeater.cc:511] Master 127.20.178.126:41855 requested a full tablet report, sending...
I20260929 03:21:21.951203 5721 ts_manager.cc:194] Registered new tserver with Master: 2dc255534e484c37b63578b25bcae523 (127.20.178.66:41135)
I20260929 03:21:21.951584 5721 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.178.66@KRBTEST.COM'} at 127.20.178.66:37549
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.178.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.178.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.67.keytab.
Entry for principal kudu/127.20.178.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.67.keytab.
I20260929 03:21:21.967057 21193 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuoaHbX/build/release/bin/kudu
/tmp/dist-test-taskuoaHbX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651852476208-21193-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651852476208-21193-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651852476208-21193-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651852476208-21193-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.178.67:0
--local_ip_for_outbound_sockets=127.20.178.67
--webserver_interface=127.20.178.67
--webserver_port=0
--tserver_master_addrs=127.20.178.126:41855
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.67.keytab
--principal=kudu/127.20.178.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.178.84:44761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651852476208-21193-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651852476208-21193-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651852476208-21193-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:21:22.047762 6051 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:21:22.047956 6051 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:21:22.047978 6051 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:21:22.049099 6051 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:21:22.049141 6051 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:21:22.049167 6051 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.178.67
I20260929 03:21:22.050535 6051 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.178.84:44761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651852476208-21193-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651852476208-21193-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.67.keytab
--principal=kudu/127.20.178.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.178.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651852476208-21193-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.178.67
--webserver_port=0
--tserver_master_addrs=127.20.178.126:41855
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6051
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.178.67
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651852476208-21193-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:21:22.050712 6051 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:21:22.050884 6051 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:21:22.055685 6057 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:21:22.055698 6056 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:21:22.055963 6051 server_base.cc:1034] running on GCE node
W20260929 03:21:22.055811 6059 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:21:22.056228 6051 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:21:22.056427 6051 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:21:22.057579 6051 hybrid_clock.cc:648] HybridClock initialized: now 1790652082057568 us; error 25 us; skew 500 ppm
Sep 29 03:21:22 dist-test-slave-c9qh krb5kdc[5676](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652082, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:21:22.059978 6051 init.cc:377] Logged in from keytab as kudu/127.20.178.67@KRBTEST.COM (short username kudu)
I20260929 03:21:22.060441 6051 webserver.cc:460] Webserver started at http://127.20.178.67:40757/ using document root <none> and password file <none>
I20260929 03:21:22.060734 6051 fs_manager.cc:359] Metadata directory not provided
I20260929 03:21:22.060784 6051 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:21:22.060930 6051 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:21:22.061993 6051 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651852476208-21193-0/minicluster-data/ts-2/data/instance:
uuid: "14b3a5a391274510951e9377357c53f0"
format_stamp: "Formatted at 2026-09-29 03:21:22 on dist-test-slave-c9qh"
server_key: "b0f7595bcd4b8a4edd61e1af784fff82"
server_key_iv: "f1852f3496e6a0d5818846e161a26dfc"
server_key_version: "encryptionkey@0"
I20260929 03:21:22.062351 6051 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651852476208-21193-0/minicluster-data/ts-2/wal/instance:
uuid: "14b3a5a391274510951e9377357c53f0"
format_stamp: "Formatted at 2026-09-29 03:21:22 on dist-test-slave-c9qh"
server_key: "b0f7595bcd4b8a4edd61e1af784fff82"
server_key_iv: "f1852f3496e6a0d5818846e161a26dfc"
server_key_version: "encryptionkey@0"
I20260929 03:21:22.063637 6051 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 03:21:22.064404 6066 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:21:22.064590 6051 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260929 03:21:22.064710 6051 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651852476208-21193-0/minicluster-data/ts-2/data,/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651852476208-21193-0/minicluster-data/ts-2/wal
uuid: "14b3a5a391274510951e9377357c53f0"
format_stamp: "Formatted at 2026-09-29 03:21:22 on dist-test-slave-c9qh"
server_key: "b0f7595bcd4b8a4edd61e1af784fff82"
server_key_iv: "f1852f3496e6a0d5818846e161a26dfc"
server_key_version: "encryptionkey@0"
I20260929 03:21:22.064790 6051 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651852476208-21193-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651852476208-21193-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651852476208-21193-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:21:22.088290 6051 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:21:22.090531 6051 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:21:22.090679 6051 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:21:22.090927 6051 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:21:22.091249 6051 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:21:22.091310 6051 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:21:22.091351 6051 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:21:22.091377 6051 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:21:22.097734 6051 rpc_server.cc:307] RPC server started. Bound to: 127.20.178.67:35391
I20260929 03:21:22.097772 6179 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.178.67:35391 every 8 connection(s)
I20260929 03:21:22.098100 6051 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651852476208-21193-0/minicluster-data/ts-2/data/info.pb
I20260929 03:21:22.101784 21193 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuoaHbX/build/release/bin/kudu as pid 6051
I20260929 03:21:22.101869 21193 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651852476208-21193-0/minicluster-data/ts-2/wal/instance
I20260929 03:21:22.102047 21193 external_mini_cluster.cc:1428] Setting key 9add7371e761a064f74bcb855265d5a8
Sep 29 03:21:22 dist-test-slave-c9qh krb5kdc[5676](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652082, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.67@KRBTEST.COM for kudu/127.20.178.126@KRBTEST.COM
I20260929 03:21:22.107503 5761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:22.098838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.67:56399 (local address 127.20.178.126:41855)
0929 03:21:22.098923 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0929 03:21:22.098925 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:21:22.099073 (+ 148us) server_negotiation.cc:407] Connection header received
0929 03:21:22.099588 (+ 515us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:21:22.099589 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:21:22.099599 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:21:22.099651 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:21:22.100792 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:22.101449 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:22.102604 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:22.102826 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:22.102916 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:21:22.104931 (+ 2015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:21:22.104932 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:21:22.104933 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:21:22.104941 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:21:22.106129 (+ 1188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:21:22.106529 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:21:22.106530 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:21:22.106531 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:21:22.106594 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:21:22.106844 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:21:22.106845 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:21:22.106845 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:21:22.106960 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:21:22.107039 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0929 03:21:22.107332 (+ 293us) server_negotiation.cc:299] Negotiation successful
0929 03:21:22.107411 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 03:21:22.107697 6182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:22.098781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:41855 (local address 127.20.178.67:56399)
0929 03:21:22.098952 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0929 03:21:22.098986 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0929 03:21:22.099502 (+ 516us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:21:22.099703 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:21:22.099706 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:21:22.100403 (+ 697us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:21:22.100683 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:21:22.100690 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:21:22.101526 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:21:22.101527 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:21:22.102499 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:21:22.102502 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:21:22.102926 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:21:22.102927 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:21:22.103065 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:21:22.103486 (+ 421us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:21:22.103494 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:21:22.104838 (+ 1344us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:21:22.106232 (+ 1394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:21:22.106238 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:21:22.106240 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:21:22.106451 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:21:22.106663 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:21:22.106664 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:21:22.106664 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:21:22.106774 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:21:22.107029 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:21:22.107036 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:21:22.107172 (+ 136us) client_negotiation.cc:769] Sending connection context
0929 03:21:22.107310 (+ 138us) client_negotiation.cc:240] Negotiation successful
0929 03:21:22.107466 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":110,"threads_started":1}
I20260929 03:21:22.108186 6180 heartbeater.cc:347] Connected to a master server at 127.20.178.126:41855
I20260929 03:21:22.108285 6180 heartbeater.cc:464] Registering TS with master...
I20260929 03:21:22.108457 6180 heartbeater.cc:511] Master 127.20.178.126:41855 requested a full tablet report, sending...
I20260929 03:21:22.108862 5721 ts_manager.cc:194] Registered new tserver with Master: 14b3a5a391274510951e9377357c53f0 (127.20.178.67:35391)
I20260929 03:21:22.109186 5721 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.178.67@KRBTEST.COM'} at 127.20.178.67:56399
I20260929 03:21:22.113562 21193 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 03:21:22.120731 5761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:22.114852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55886 (local address 127.20.178.126:41855)
0929 03:21:22.114907 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0929 03:21:22.114908 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 03:21:22.115002 (+ 94us) server_negotiation.cc:407] Connection header received
0929 03:21:22.115050 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:21:22.115051 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:21:22.115059 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:21:22.115107 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:21:22.115613 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:22.116079 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:22.116836 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:22.117008 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:22.117058 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:21:22.118093 (+ 1035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:21:22.118094 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:21:22.118094 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:21:22.118100 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:21:22.119786 (+ 1686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:21:22.120120 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:21:22.120121 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:21:22.120121 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:21:22.120168 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:21:22.120292 (+ 124us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:21:22.120292 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:21:22.120292 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:21:22.120412 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:21:22.120485 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 03:21:22.120549 (+ 64us) server_negotiation.cc:299] Negotiation successful
0929 03:21:22.120601 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260929 03:21:22.122172 5721 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55886:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 03:21:22.123689 5721 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 03:21:22.130575 6195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:22.126470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46692 (local address 127.20.178.65:43363)
0929 03:21:22.126637 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0929 03:21:22.126642 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:21:22.126652 (+ 10us) server_negotiation.cc:407] Connection header received
0929 03:21:22.126691 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:21:22.126694 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:21:22.126710 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:21:22.126934 (+ 224us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:21:22.127005 (+ 71us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:22.128061 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:22.129470 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:22.130205 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:22.130284 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:21:22.130342 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0929 03:21:22.130481 (+ 139us) server_negotiation.cc:299] Negotiation successful
0929 03:21:22.130512 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":89,"threads_started":1}
I20260929 03:21:22.130604 6192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:22.126028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.65:43363 (local address 127.0.0.1:46692)
0929 03:21:22.126361 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0929 03:21:22.126369 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0929 03:21:22.126402 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:21:22.126806 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:21:22.126807 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:21:22.126808 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:21:22.126858 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:21:22.126860 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:21:22.128186 (+ 1326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:21:22.128187 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:21:22.129373 (+ 1186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:21:22.129375 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:21:22.130292 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:21:22.130293 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:21:22.130378 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:21:22.130391 (+ 13us) client_negotiation.cc:769] Sending connection context
0929 03:21:22.130470 (+ 79us) client_negotiation.cc:240] Negotiation successful
0929 03:21:22.130518 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":318,"mutex_wait_us":50,"thread_start_us":98,"threads_started":1}
I20260929 03:21:22.131039 6191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:22.126018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.66:41135 (local address 127.0.0.1:46064)
0929 03:21:22.126157 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0929 03:21:22.126196 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 03:21:22.126261 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:21:22.127030 (+ 769us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:21:22.127031 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:21:22.127034 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:21:22.127080 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:21:22.127081 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:21:22.128028 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:21:22.128029 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:21:22.129372 (+ 1343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:21:22.129375 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:21:22.130812 (+ 1437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:21:22.130813 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:21:22.130880 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:21:22.130884 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 03:21:22.130933 (+ 49us) client_negotiation.cc:240] Negotiation successful
0929 03:21:22.130970 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"mutex_wait_us":51,"thread_start_us":45,"threads_started":1}
I20260929 03:21:22.131211 6193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:22.126028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.67:35391 (local address 127.0.0.1:60628)
0929 03:21:22.126161 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0929 03:21:22.126196 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0929 03:21:22.126261 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:21:22.126782 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:21:22.126785 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:21:22.126787 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:21:22.126859 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:21:22.126860 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:21:22.128170 (+ 1310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:21:22.128171 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:21:22.129991 (+ 1820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:21:22.129993 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:21:22.130998 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:21:22.130999 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:21:22.131052 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:21:22.131056 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 03:21:22.131113 (+ 57us) client_negotiation.cc:240] Negotiation successful
0929 03:21:22.131152 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"mutex_wait_us":75,"thread_start_us":97,"threads_started":1}
I20260929 03:21:22.131356 6196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:22.126671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46064 (local address 127.20.178.66:41135)
0929 03:21:22.126877 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0929 03:21:22.126882 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:21:22.126892 (+ 10us) server_negotiation.cc:407] Connection header received
0929 03:21:22.126932 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:21:22.126935 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:21:22.126951 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:21:22.127020 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:21:22.127152 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:22.127928 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:22.129490 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:22.130201 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:22.130284 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:21:22.130343 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0929 03:21:22.131193 (+ 850us) server_negotiation.cc:299] Negotiation successful
0929 03:21:22.131242 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":96,"threads_started":1}
I20260929 03:21:22.131402 5844 tablet_service.cc:1467] Processing CreateTablet for tablet 1338fb80bb5a4a50ac511b5b9bee4161 (DEFAULT_TABLE table=test-table [id=ffe0eda2f90a4bb0b3c2d8955061e690]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:21:22.131613 6194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:22.126263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60628 (local address 127.20.178.67:35391)
0929 03:21:22.126451 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0929 03:21:22.126456 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:21:22.126467 (+ 11us) server_negotiation.cc:407] Connection header received
0929 03:21:22.126508 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:21:22.126511 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:21:22.126528 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:21:22.126620 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:21:22.126944 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:22.128081 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:22.130086 (+ 2005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:22.130846 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:22.130911 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:21:22.130967 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0929 03:21:22.131376 (+ 409us) server_negotiation.cc:299] Negotiation successful
0929 03:21:22.131537 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":102,"threads_started":1}
I20260929 03:21:22.131773 5844 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1338fb80bb5a4a50ac511b5b9bee4161. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:21:22.132195 6114 tablet_service.cc:1467] Processing CreateTablet for tablet 1338fb80bb5a4a50ac511b5b9bee4161 (DEFAULT_TABLE table=test-table [id=ffe0eda2f90a4bb0b3c2d8955061e690]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:21:22.132483 6114 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1338fb80bb5a4a50ac511b5b9bee4161. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:21:22.133033 5980 tablet_service.cc:1467] Processing CreateTablet for tablet 1338fb80bb5a4a50ac511b5b9bee4161 (DEFAULT_TABLE table=test-table [id=ffe0eda2f90a4bb0b3c2d8955061e690]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:21:22.133364 5980 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1338fb80bb5a4a50ac511b5b9bee4161. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:21:22.134639 6197 tablet_bootstrap.cc:492] T 1338fb80bb5a4a50ac511b5b9bee4161 P 42cfed0d018c4ec58518a3d749e62c70: Bootstrap starting.
I20260929 03:21:22.135264 6197 tablet_bootstrap.cc:654] T 1338fb80bb5a4a50ac511b5b9bee4161 P 42cfed0d018c4ec58518a3d749e62c70: Neither blocks nor log segments found. Creating new log.
I20260929 03:21:22.135895 6198 tablet_bootstrap.cc:492] T 1338fb80bb5a4a50ac511b5b9bee4161 P 14b3a5a391274510951e9377357c53f0: Bootstrap starting.
I20260929 03:21:22.135934 6197 log.cc:824] T 1338fb80bb5a4a50ac511b5b9bee4161 P 42cfed0d018c4ec58518a3d749e62c70: Log is configured to *not* fsync() on all Append() calls
I20260929 03:21:22.136529 6198 tablet_bootstrap.cc:654] T 1338fb80bb5a4a50ac511b5b9bee4161 P 14b3a5a391274510951e9377357c53f0: Neither blocks nor log segments found. Creating new log.
I20260929 03:21:22.136746 6197 tablet_bootstrap.cc:492] T 1338fb80bb5a4a50ac511b5b9bee4161 P 42cfed0d018c4ec58518a3d749e62c70: No bootstrap required, opened a new log
I20260929 03:21:22.136828 6197 ts_tablet_manager.cc:1397] T 1338fb80bb5a4a50ac511b5b9bee4161 P 42cfed0d018c4ec58518a3d749e62c70: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 03:21:22.137132 6198 log.cc:824] T 1338fb80bb5a4a50ac511b5b9bee4161 P 14b3a5a391274510951e9377357c53f0: Log is configured to *not* fsync() on all Append() calls
I20260929 03:21:22.137851 6198 tablet_bootstrap.cc:492] T 1338fb80bb5a4a50ac511b5b9bee4161 P 14b3a5a391274510951e9377357c53f0: No bootstrap required, opened a new log
I20260929 03:21:22.137918 6198 ts_tablet_manager.cc:1397] T 1338fb80bb5a4a50ac511b5b9bee4161 P 14b3a5a391274510951e9377357c53f0: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 03:21:22.138931 6197 raft_consensus.cc:348] T 1338fb80bb5a4a50ac511b5b9bee4161 P 42cfed0d018c4ec58518a3d749e62c70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14b3a5a391274510951e9377357c53f0" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 35391 } } peers { permanent_uuid: "42cfed0d018c4ec58518a3d749e62c70" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 43363 } } peers { permanent_uuid: "2dc255534e484c37b63578b25bcae523" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 41135 } }
I20260929 03:21:22.139110 6197 raft_consensus.cc:374] T 1338fb80bb5a4a50ac511b5b9bee4161 P 42cfed0d018c4ec58518a3d749e62c70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:21:22.139161 6197 raft_consensus.cc:729] T 1338fb80bb5a4a50ac511b5b9bee4161 P 42cfed0d018c4ec58518a3d749e62c70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42cfed0d018c4ec58518a3d749e62c70, State: Initialized, Role: FOLLOWER
I20260929 03:21:22.139281 6197 consensus_queue.cc:260] T 1338fb80bb5a4a50ac511b5b9bee4161 P 42cfed0d018c4ec58518a3d749e62c70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14b3a5a391274510951e9377357c53f0" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 35391 } } peers { permanent_uuid: "42cfed0d018c4ec58518a3d749e62c70" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 43363 } } peers { permanent_uuid: "2dc255534e484c37b63578b25bcae523" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 41135 } }
I20260929 03:21:22.139555 6197 ts_tablet_manager.cc:1428] T 1338fb80bb5a4a50ac511b5b9bee4161 P 42cfed0d018c4ec58518a3d749e62c70: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260929 03:21:22.139582 5912 heartbeater.cc:503] Master 127.20.178.126:41855 was elected leader, sending a full tablet report...
I20260929 03:21:22.139753 6200 tablet_bootstrap.cc:492] T 1338fb80bb5a4a50ac511b5b9bee4161 P 2dc255534e484c37b63578b25bcae523: Bootstrap starting.
I20260929 03:21:22.139912 6198 raft_consensus.cc:348] T 1338fb80bb5a4a50ac511b5b9bee4161 P 14b3a5a391274510951e9377357c53f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14b3a5a391274510951e9377357c53f0" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 35391 } } peers { permanent_uuid: "42cfed0d018c4ec58518a3d749e62c70" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 43363 } } peers { permanent_uuid: "2dc255534e484c37b63578b25bcae523" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 41135 } }
I20260929 03:21:22.140089 6198 raft_consensus.cc:374] T 1338fb80bb5a4a50ac511b5b9bee4161 P 14b3a5a391274510951e9377357c53f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:21:22.140133 6198 raft_consensus.cc:729] T 1338fb80bb5a4a50ac511b5b9bee4161 P 14b3a5a391274510951e9377357c53f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14b3a5a391274510951e9377357c53f0, State: Initialized, Role: FOLLOWER
I20260929 03:21:22.140273 6198 consensus_queue.cc:260] T 1338fb80bb5a4a50ac511b5b9bee4161 P 14b3a5a391274510951e9377357c53f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14b3a5a391274510951e9377357c53f0" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 35391 } } peers { permanent_uuid: "42cfed0d018c4ec58518a3d749e62c70" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 43363 } } peers { permanent_uuid: "2dc255534e484c37b63578b25bcae523" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 41135 } }
I20260929 03:21:22.140385 6200 tablet_bootstrap.cc:654] T 1338fb80bb5a4a50ac511b5b9bee4161 P 2dc255534e484c37b63578b25bcae523: Neither blocks nor log segments found. Creating new log.
I20260929 03:21:22.140569 6198 ts_tablet_manager.cc:1428] T 1338fb80bb5a4a50ac511b5b9bee4161 P 14b3a5a391274510951e9377357c53f0: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260929 03:21:22.140733 6180 heartbeater.cc:503] Master 127.20.178.126:41855 was elected leader, sending a full tablet report...
I20260929 03:21:22.140998 6200 log.cc:824] T 1338fb80bb5a4a50ac511b5b9bee4161 P 2dc255534e484c37b63578b25bcae523: Log is configured to *not* fsync() on all Append() calls
I20260929 03:21:22.141747 6200 tablet_bootstrap.cc:492] T 1338fb80bb5a4a50ac511b5b9bee4161 P 2dc255534e484c37b63578b25bcae523: No bootstrap required, opened a new log
I20260929 03:21:22.141845 6200 ts_tablet_manager.cc:1397] T 1338fb80bb5a4a50ac511b5b9bee4161 P 2dc255534e484c37b63578b25bcae523: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260929 03:21:22.143150 6200 raft_consensus.cc:348] T 1338fb80bb5a4a50ac511b5b9bee4161 P 2dc255534e484c37b63578b25bcae523 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14b3a5a391274510951e9377357c53f0" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 35391 } } peers { permanent_uuid: "42cfed0d018c4ec58518a3d749e62c70" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 43363 } } peers { permanent_uuid: "2dc255534e484c37b63578b25bcae523" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 41135 } }
I20260929 03:21:22.143375 6200 raft_consensus.cc:374] T 1338fb80bb5a4a50ac511b5b9bee4161 P 2dc255534e484c37b63578b25bcae523 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:21:22.143414 6200 raft_consensus.cc:729] T 1338fb80bb5a4a50ac511b5b9bee4161 P 2dc255534e484c37b63578b25bcae523 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2dc255534e484c37b63578b25bcae523, State: Initialized, Role: FOLLOWER
I20260929 03:21:22.143525 6200 consensus_queue.cc:260] T 1338fb80bb5a4a50ac511b5b9bee4161 P 2dc255534e484c37b63578b25bcae523 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14b3a5a391274510951e9377357c53f0" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 35391 } } peers { permanent_uuid: "42cfed0d018c4ec58518a3d749e62c70" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 43363 } } peers { permanent_uuid: "2dc255534e484c37b63578b25bcae523" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 41135 } }
I20260929 03:21:22.143837 6200 ts_tablet_manager.cc:1428] T 1338fb80bb5a4a50ac511b5b9bee4161 P 2dc255534e484c37b63578b25bcae523: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260929 03:21:22.143882 6046 heartbeater.cc:503] Master 127.20.178.126:41855 was elected leader, sending a full tablet report...
I20260929 03:21:22.145493 6203 raft_consensus.cc:482] T 1338fb80bb5a4a50ac511b5b9bee4161 P 14b3a5a391274510951e9377357c53f0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:21:22.145566 6203 raft_consensus.cc:504] T 1338fb80bb5a4a50ac511b5b9bee4161 P 14b3a5a391274510951e9377357c53f0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14b3a5a391274510951e9377357c53f0" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 35391 } } peers { permanent_uuid: "42cfed0d018c4ec58518a3d749e62c70" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 43363 } } peers { permanent_uuid: "2dc255534e484c37b63578b25bcae523" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 41135 } }
I20260929 03:21:22.145799 6203 leader_election.cc:288] T 1338fb80bb5a4a50ac511b5b9bee4161 P 14b3a5a391274510951e9377357c53f0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 42cfed0d018c4ec58518a3d749e62c70 (127.20.178.65:43363), 2dc255534e484c37b63578b25bcae523 (127.20.178.66:41135)
I20260929 03:21:22.148535 6195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:22.146090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.67:43835 (local address 127.20.178.65:43363)
0929 03:21:22.146137 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0929 03:21:22.146139 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:21:22.146150 (+ 11us) server_negotiation.cc:407] Connection header received
0929 03:21:22.146175 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:21:22.146176 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:21:22.146185 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:21:22.146240 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:21:22.146380 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:22.146903 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:22.147821 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:22.148205 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:22.148261 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:21:22.148268 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0929 03:21:22.148424 (+ 156us) server_negotiation.cc:299] Negotiation successful
0929 03:21:22.148470 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260929 03:21:22.148535 6182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:22.145950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.65:43363 (local address 127.20.178.67:43835)
0929 03:21:22.146021 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0929 03:21:22.146034 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 03:21:22.146082 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:21:22.146249 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:21:22.146250 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:21:22.146252 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:21:22.146321 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:21:22.146323 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:21:22.146981 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:21:22.146983 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:21:22.147734 (+ 751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:21:22.147736 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:21:22.148283 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:21:22.148284 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:21:22.148362 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:21:22.148365 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 03:21:22.148416 (+ 51us) client_negotiation.cc:240] Negotiation successful
0929 03:21:22.148460 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49,"mutex_wait_us":19}
I20260929 03:21:22.148587 6196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:22.146081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.67:58239 (local address 127.20.178.66:41135)
0929 03:21:22.146138 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0929 03:21:22.146140 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:21:22.146200 (+ 60us) server_negotiation.cc:407] Connection header received
0929 03:21:22.146310 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:21:22.146311 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:21:22.146332 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:21:22.146385 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:21:22.146495 (+ 110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:22.146920 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:22.147872 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:22.148225 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:22.148316 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:21:22.148332 (+ 16us) server_negotiation.cc:1035] Waiting for connection context
0929 03:21:22.148485 (+ 153us) server_negotiation.cc:299] Negotiation successful
0929 03:21:22.148519 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260929 03:21:22.148931 5866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1338fb80bb5a4a50ac511b5b9bee4161" candidate_uuid: "14b3a5a391274510951e9377357c53f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42cfed0d018c4ec58518a3d749e62c70" is_pre_election: true
I20260929 03:21:22.149103 5866 raft_consensus.cc:2454] T 1338fb80bb5a4a50ac511b5b9bee4161 P 42cfed0d018c4ec58518a3d749e62c70 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 14b3a5a391274510951e9377357c53f0 in term 0.
I20260929 03:21:22.149138 6206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:22.145950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.66:41135 (local address 127.20.178.67:58239)
0929 03:21:22.146140 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0929 03:21:22.146150 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 03:21:22.146211 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:21:22.146384 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:21:22.146385 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:21:22.146387 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:21:22.146446 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:21:22.146447 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:21:22.147025 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:21:22.147030 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:21:22.147756 (+ 726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:21:22.147758 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:21:22.148328 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:21:22.148329 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:21:22.148423 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:21:22.148427 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 03:21:22.148481 (+ 54us) client_negotiation.cc:240] Negotiation successful
0929 03:21:22.148517 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"mutex_wait_us":25,"thread_start_us":78,"threads_started":1}
I20260929 03:21:22.149318 6068 leader_election.cc:302] T 1338fb80bb5a4a50ac511b5b9bee4161 P 14b3a5a391274510951e9377357c53f0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14b3a5a391274510951e9377357c53f0, 42cfed0d018c4ec58518a3d749e62c70; no voters:
I20260929 03:21:22.149456 6203 raft_consensus.cc:2781] T 1338fb80bb5a4a50ac511b5b9bee4161 P 14b3a5a391274510951e9377357c53f0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 03:21:22.149506 6203 raft_consensus.cc:482] T 1338fb80bb5a4a50ac511b5b9bee4161 P 14b3a5a391274510951e9377357c53f0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 03:21:22.149453 6000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1338fb80bb5a4a50ac511b5b9bee4161" candidate_uuid: "14b3a5a391274510951e9377357c53f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2dc255534e484c37b63578b25bcae523" is_pre_election: true
I20260929 03:21:22.149539 6203 raft_consensus.cc:3037] T 1338fb80bb5a4a50ac511b5b9bee4161 P 14b3a5a391274510951e9377357c53f0 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:21:22.149583 6000 raft_consensus.cc:2454] T 1338fb80bb5a4a50ac511b5b9bee4161 P 2dc255534e484c37b63578b25bcae523 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 14b3a5a391274510951e9377357c53f0 in term 0.
I20260929 03:21:22.150247 6203 raft_consensus.cc:504] T 1338fb80bb5a4a50ac511b5b9bee4161 P 14b3a5a391274510951e9377357c53f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14b3a5a391274510951e9377357c53f0" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 35391 } } peers { permanent_uuid: "42cfed0d018c4ec58518a3d749e62c70" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 43363 } } peers { permanent_uuid: "2dc255534e484c37b63578b25bcae523" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 41135 } }
I20260929 03:21:22.150388 6203 leader_election.cc:288] T 1338fb80bb5a4a50ac511b5b9bee4161 P 14b3a5a391274510951e9377357c53f0 [CANDIDATE]: Term 1 election: Requested vote from peers 42cfed0d018c4ec58518a3d749e62c70 (127.20.178.65:43363), 2dc255534e484c37b63578b25bcae523 (127.20.178.66:41135)
I20260929 03:21:22.150506 5866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1338fb80bb5a4a50ac511b5b9bee4161" candidate_uuid: "14b3a5a391274510951e9377357c53f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42cfed0d018c4ec58518a3d749e62c70"
I20260929 03:21:22.150538 6000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1338fb80bb5a4a50ac511b5b9bee4161" candidate_uuid: "14b3a5a391274510951e9377357c53f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2dc255534e484c37b63578b25bcae523"
I20260929 03:21:22.150592 6000 raft_consensus.cc:3037] T 1338fb80bb5a4a50ac511b5b9bee4161 P 2dc255534e484c37b63578b25bcae523 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:21:22.150584 5866 raft_consensus.cc:3037] T 1338fb80bb5a4a50ac511b5b9bee4161 P 42cfed0d018c4ec58518a3d749e62c70 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:21:22.151425 5866 raft_consensus.cc:2454] T 1338fb80bb5a4a50ac511b5b9bee4161 P 42cfed0d018c4ec58518a3d749e62c70 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 14b3a5a391274510951e9377357c53f0 in term 1.
I20260929 03:21:22.151425 6000 raft_consensus.cc:2454] T 1338fb80bb5a4a50ac511b5b9bee4161 P 2dc255534e484c37b63578b25bcae523 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 14b3a5a391274510951e9377357c53f0 in term 1.
I20260929 03:21:22.151605 6070 leader_election.cc:302] T 1338fb80bb5a4a50ac511b5b9bee4161 P 14b3a5a391274510951e9377357c53f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14b3a5a391274510951e9377357c53f0, 2dc255534e484c37b63578b25bcae523; no voters:
I20260929 03:21:22.151700 6203 raft_consensus.cc:2781] T 1338fb80bb5a4a50ac511b5b9bee4161 P 14b3a5a391274510951e9377357c53f0 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:21:22.151836 6203 raft_consensus.cc:686] T 1338fb80bb5a4a50ac511b5b9bee4161 P 14b3a5a391274510951e9377357c53f0 [term 1 LEADER]: Becoming Leader. State: Replica: 14b3a5a391274510951e9377357c53f0, State: Running, Role: LEADER
I20260929 03:21:22.151929 6203 consensus_queue.cc:237] T 1338fb80bb5a4a50ac511b5b9bee4161 P 14b3a5a391274510951e9377357c53f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14b3a5a391274510951e9377357c53f0" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 35391 } } peers { permanent_uuid: "42cfed0d018c4ec58518a3d749e62c70" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 43363 } } peers { permanent_uuid: "2dc255534e484c37b63578b25bcae523" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 41135 } }
I20260929 03:21:22.152709 5721 catalog_manager.cc:5495] T 1338fb80bb5a4a50ac511b5b9bee4161 P 14b3a5a391274510951e9377357c53f0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 14b3a5a391274510951e9377357c53f0 (127.20.178.67). New cstate: current_term: 1 leader_uuid: "14b3a5a391274510951e9377357c53f0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14b3a5a391274510951e9377357c53f0" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 35391 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "42cfed0d018c4ec58518a3d749e62c70" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 43363 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2dc255534e484c37b63578b25bcae523" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 41135 } health_report { overall_health: UNKNOWN } } }
I20260929 03:21:22.159380 6194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:22.157073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60636 (local address 127.20.178.67:35391)
0929 03:21:22.157126 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0929 03:21:22.157128 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:21:22.157134 (+ 6us) server_negotiation.cc:407] Connection header received
0929 03:21:22.157159 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:21:22.157160 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:21:22.157173 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:21:22.157232 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:21:22.157367 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:22.157794 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:22.158482 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:22.158652 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:22.158700 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:21:22.158912 (+ 212us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:21:22.159039 (+ 127us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:21:22.159138 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0929 03:21:22.159195 (+ 57us) server_negotiation.cc:299] Negotiation successful
0929 03:21:22.159218 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260929 03:21:22.161176 6000 raft_consensus.cc:1261] T 1338fb80bb5a4a50ac511b5b9bee4161 P 2dc255534e484c37b63578b25bcae523 [term 1 FOLLOWER]: Refusing update from remote peer 14b3a5a391274510951e9377357c53f0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:21:22.161192 5866 raft_consensus.cc:1261] T 1338fb80bb5a4a50ac511b5b9bee4161 P 42cfed0d018c4ec58518a3d749e62c70 [term 1 FOLLOWER]: Refusing update from remote peer 14b3a5a391274510951e9377357c53f0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:21:22.161523 6207 consensus_queue.cc:1035] T 1338fb80bb5a4a50ac511b5b9bee4161 P 14b3a5a391274510951e9377357c53f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42cfed0d018c4ec58518a3d749e62c70" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 43363 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:21:22.161635 6203 consensus_queue.cc:1035] T 1338fb80bb5a4a50ac511b5b9bee4161 P 14b3a5a391274510951e9377357c53f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2dc255534e484c37b63578b25bcae523" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 41135 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:21:22.163800 6210 mvcc.cc:205] Tried to move back new op lower bound from 7334510928529539072 to 7334510928494907392. Current Snapshot: MvccSnapshot[applied={T|T < 7334510928529539072}]
I20260929 03:21:22.167058 5720 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55886:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:21:22.167141 5720 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55886:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:21:22.167936 5720 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2c67e8a370c547198280b321887ff8cf: Sending DeleteTablet for 3 replicas of tablet 1338fb80bb5a4a50ac511b5b9bee4161
I20260929 03:21:22.168277 5844 tablet_service.cc:1514] Processing DeleteTablet for tablet 1338fb80bb5a4a50ac511b5b9bee4161 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:21:22 UTC) from {username='kudu', principal='kudu/127.20.178.126@KRBTEST.COM'} at 127.0.0.1:46692
I20260929 03:21:22.168370 5980 tablet_service.cc:1514] Processing DeleteTablet for tablet 1338fb80bb5a4a50ac511b5b9bee4161 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:21:22 UTC) from {username='kudu', principal='kudu/127.20.178.126@KRBTEST.COM'} at 127.0.0.1:46064
I20260929 03:21:22.168479 6114 tablet_service.cc:1514] Processing DeleteTablet for tablet 1338fb80bb5a4a50ac511b5b9bee4161 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:21:22 UTC) from {username='kudu', principal='kudu/127.20.178.126@KRBTEST.COM'} at 127.0.0.1:60628
I20260929 03:21:22.168663 6221 tablet_replica.cc:330] T 1338fb80bb5a4a50ac511b5b9bee4161 P 2dc255534e484c37b63578b25bcae523: stopping tablet replica
I20260929 03:21:22.168774 6220 tablet_replica.cc:330] T 1338fb80bb5a4a50ac511b5b9bee4161 P 42cfed0d018c4ec58518a3d749e62c70: stopping tablet replica
I20260929 03:21:22.168808 6221 raft_consensus.cc:2229] T 1338fb80bb5a4a50ac511b5b9bee4161 P 2dc255534e484c37b63578b25bcae523 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:21:22.168881 6220 raft_consensus.cc:2229] T 1338fb80bb5a4a50ac511b5b9bee4161 P 42cfed0d018c4ec58518a3d749e62c70 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:21:22.168897 21193 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuoaHbX/build/release/bin/kudu with pid 5783
I20260929 03:21:22.168989 6220 raft_consensus.cc:2258] T 1338fb80bb5a4a50ac511b5b9bee4161 P 42cfed0d018c4ec58518a3d749e62c70 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:21:22.169080 6221 raft_consensus.cc:2258] T 1338fb80bb5a4a50ac511b5b9bee4161 P 2dc255534e484c37b63578b25bcae523 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:21:22.169153 6222 tablet_replica.cc:330] T 1338fb80bb5a4a50ac511b5b9bee4161 P 14b3a5a391274510951e9377357c53f0: stopping tablet replica
I20260929 03:21:22.169240 6222 raft_consensus.cc:2229] T 1338fb80bb5a4a50ac511b5b9bee4161 P 14b3a5a391274510951e9377357c53f0 [term 1 LEADER]: Raft consensus shutting down.
I20260929 03:21:22.169580 6222 raft_consensus.cc:2258] T 1338fb80bb5a4a50ac511b5b9bee4161 P 14b3a5a391274510951e9377357c53f0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:21:22.169776 6221 ts_tablet_manager.cc:1905] T 1338fb80bb5a4a50ac511b5b9bee4161 P 2dc255534e484c37b63578b25bcae523: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:21:22.170013 6222 ts_tablet_manager.cc:1905] T 1338fb80bb5a4a50ac511b5b9bee4161 P 14b3a5a391274510951e9377357c53f0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:21:22.171861 6221 ts_tablet_manager.cc:1918] T 1338fb80bb5a4a50ac511b5b9bee4161 P 2dc255534e484c37b63578b25bcae523: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:21:22.171880 6222 ts_tablet_manager.cc:1918] T 1338fb80bb5a4a50ac511b5b9bee4161 P 14b3a5a391274510951e9377357c53f0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:21:22.171937 6222 log.cc:1196] T 1338fb80bb5a4a50ac511b5b9bee4161 P 14b3a5a391274510951e9377357c53f0: Deleting WAL directory at /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651852476208-21193-0/minicluster-data/ts-2/wal/wals/1338fb80bb5a4a50ac511b5b9bee4161
I20260929 03:21:22.172045 6221 log.cc:1196] T 1338fb80bb5a4a50ac511b5b9bee4161 P 2dc255534e484c37b63578b25bcae523: Deleting WAL directory at /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651852476208-21193-0/minicluster-data/ts-1/wal/wals/1338fb80bb5a4a50ac511b5b9bee4161
I20260929 03:21:22.172224 6222 ts_tablet_manager.cc:1939] T 1338fb80bb5a4a50ac511b5b9bee4161 P 14b3a5a391274510951e9377357c53f0: Deleting consensus metadata
I20260929 03:21:22.172429 6221 ts_tablet_manager.cc:1939] T 1338fb80bb5a4a50ac511b5b9bee4161 P 2dc255534e484c37b63578b25bcae523: Deleting consensus metadata
I20260929 03:21:22.172621 5705 catalog_manager.cc:4841] TS 14b3a5a391274510951e9377357c53f0 (127.20.178.67:35391): tablet 1338fb80bb5a4a50ac511b5b9bee4161 (table test-table [id=ffe0eda2f90a4bb0b3c2d8955061e690]) successfully deleted
I20260929 03:21:22.173010 5708 catalog_manager.cc:4841] TS 2dc255534e484c37b63578b25bcae523 (127.20.178.66:41135): tablet 1338fb80bb5a4a50ac511b5b9bee4161 (table test-table [id=ffe0eda2f90a4bb0b3c2d8955061e690]) successfully deleted
W20260929 03:21:22.176043 5706 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.20.178.65:43363 (error 108)
I20260929 03:21:22.176273 21193 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuoaHbX/build/release/bin/kudu with pid 5917
I20260929 03:21:22.176831 6191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:22.176435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.65:43363 (local address 127.0.0.1:46696)
0929 03:21:22.176761 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0929 03:21:22.176797 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.178.65:43363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":303}
W20260929 03:21:22.177148 5706 catalog_manager.cc:4584] TS 42cfed0d018c4ec58518a3d749e62c70 (127.20.178.65:43363): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1338fb80bb5a4a50ac511b5b9bee4161: Network error: Client connection negotiation failed: client connection to 127.20.178.65:43363: connect: Connection refused (error 111)
I20260929 03:21:22.180943 21193 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuoaHbX/build/release/bin/kudu with pid 6051
I20260929 03:21:22.185290 21193 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuoaHbX/build/release/bin/kudu with pid 5689
2026-09-29T03:21:22Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (1911 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-29T03:21:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:21:22Z Disabled control of system clock
I20260929 03:21:22.212097 21193 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuoaHbX/build/release/bin/kudu
/tmp/dist-test-taskuoaHbX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790651852476208-21193-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790651852476208-21193-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790651852476208-21193-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790651852476208-21193-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.178.126:44805
--webserver_interface=127.20.178.126
--webserver_port=0
--builtin_ntp_servers=127.20.178.84:43541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.178.126:44805
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 03:21:22.291762 6227 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:21:22.291975 6227 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:21:22.291996 6227 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:21:22.293179 6227 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:21:22.293226 6227 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:21:22.293241 6227 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:21:22.293253 6227 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:21:22.293265 6227 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:21:22.294544 6227 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.178.84:43541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790651852476208-21193-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790651852476208-21193-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.178.126:44805
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.178.126:44805
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790651852476208-21193-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.178.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6227
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790651852476208-21193-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:21:22.294744 6227 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:21:22.294968 6227 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:21:22.299588 6233 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:21:22.299628 6235 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:21:22.299624 6232 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:21:22.299830 6227 server_base.cc:1034] running on GCE node
I20260929 03:21:22.300148 6227 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:21:22.300469 6227 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:21:22.301618 6227 hybrid_clock.cc:648] HybridClock initialized: now 1790652082301606 us; error 34 us; skew 500 ppm
I20260929 03:21:22.302847 6227 webserver.cc:460] Webserver started at http://127.20.178.126:35107/ using document root <none> and password file <none>
I20260929 03:21:22.303087 6227 fs_manager.cc:359] Metadata directory not provided
I20260929 03:21:22.303135 6227 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:21:22.303318 6227 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:21:22.304346 6227 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790651852476208-21193-0/minicluster-data/master-0/data/instance:
uuid: "322d1455dc2247a2ad27f51d40ec47a9"
format_stamp: "Formatted at 2026-09-29 03:21:22 on dist-test-slave-c9qh"
server_key: "19af57584ebe4986994b07e4e1f3496c"
server_key_iv: "572ce23bcc84e10a08dff35624c3326e"
server_key_version: "encryptionkey@0"
I20260929 03:21:22.304736 6227 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790651852476208-21193-0/minicluster-data/master-0/wal/instance:
uuid: "322d1455dc2247a2ad27f51d40ec47a9"
format_stamp: "Formatted at 2026-09-29 03:21:22 on dist-test-slave-c9qh"
server_key: "19af57584ebe4986994b07e4e1f3496c"
server_key_iv: "572ce23bcc84e10a08dff35624c3326e"
server_key_version: "encryptionkey@0"
I20260929 03:21:22.305986 6227 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 03:21:22.306712 6241 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:21:22.306896 6227 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 03:21:22.306963 6227 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790651852476208-21193-0/minicluster-data/master-0/data,/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790651852476208-21193-0/minicluster-data/master-0/wal
uuid: "322d1455dc2247a2ad27f51d40ec47a9"
format_stamp: "Formatted at 2026-09-29 03:21:22 on dist-test-slave-c9qh"
server_key: "19af57584ebe4986994b07e4e1f3496c"
server_key_iv: "572ce23bcc84e10a08dff35624c3326e"
server_key_version: "encryptionkey@0"
I20260929 03:21:22.307046 6227 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790651852476208-21193-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790651852476208-21193-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790651852476208-21193-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:21:22.318508 6227 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:21:22.318841 6227 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:21:22.319012 6227 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:21:22.323899 6227 rpc_server.cc:307] RPC server started. Bound to: 127.20.178.126:44805
I20260929 03:21:22.323966 6293 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.178.126:44805 every 8 connection(s)
I20260929 03:21:22.324318 6227 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790651852476208-21193-0/minicluster-data/master-0/data/info.pb
I20260929 03:21:22.325551 6294 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:21:22.326329 21193 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuoaHbX/build/release/bin/kudu as pid 6227
I20260929 03:21:22.326418 21193 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790651852476208-21193-0/minicluster-data/master-0/wal/instance
I20260929 03:21:22.326610 21193 external_mini_cluster.cc:1428] Setting key 33857d72649463acb3612dcecbd96346
I20260929 03:21:22.327982 6294 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 322d1455dc2247a2ad27f51d40ec47a9: Bootstrap starting.
I20260929 03:21:22.328573 6294 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 322d1455dc2247a2ad27f51d40ec47a9: Neither blocks nor log segments found. Creating new log.
I20260929 03:21:22.328896 6294 log.cc:824] T 00000000000000000000000000000000 P 322d1455dc2247a2ad27f51d40ec47a9: Log is configured to *not* fsync() on all Append() calls
I20260929 03:21:22.329538 6294 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 322d1455dc2247a2ad27f51d40ec47a9: No bootstrap required, opened a new log
I20260929 03:21:22.331176 6294 raft_consensus.cc:348] T 00000000000000000000000000000000 P 322d1455dc2247a2ad27f51d40ec47a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "322d1455dc2247a2ad27f51d40ec47a9" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 44805 } }
I20260929 03:21:22.331331 6294 raft_consensus.cc:374] T 00000000000000000000000000000000 P 322d1455dc2247a2ad27f51d40ec47a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:21:22.331380 6294 raft_consensus.cc:729] T 00000000000000000000000000000000 P 322d1455dc2247a2ad27f51d40ec47a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 322d1455dc2247a2ad27f51d40ec47a9, State: Initialized, Role: FOLLOWER
I20260929 03:21:22.331528 6294 consensus_queue.cc:260] T 00000000000000000000000000000000 P 322d1455dc2247a2ad27f51d40ec47a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "322d1455dc2247a2ad27f51d40ec47a9" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 44805 } }
I20260929 03:21:22.331621 6294 raft_consensus.cc:388] T 00000000000000000000000000000000 P 322d1455dc2247a2ad27f51d40ec47a9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:21:22.331663 6294 raft_consensus.cc:482] T 00000000000000000000000000000000 P 322d1455dc2247a2ad27f51d40ec47a9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:21:22.331700 6294 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 322d1455dc2247a2ad27f51d40ec47a9 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:21:22.332055 6297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:22.327095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56710 (local address 127.20.178.126:44805)
0929 03:21:22.327297 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0929 03:21:22.327304 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:21:22.327332 (+ 28us) server_negotiation.cc:407] Connection header received
0929 03:21:22.327803 (+ 471us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:21:22.327809 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:21:22.327873 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:21:22.328017 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:21:22.328227 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:22.329225 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:22.330084 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:22.330494 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:22.330645 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:21:22.331343 (+ 698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:21:22.331346 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:21:22.331347 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:21:22.331547 (+ 200us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:21:22.331579 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:21:22.331589 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:21:22.331661 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0929 03:21:22.331736 (+ 75us) server_negotiation.cc:299] Negotiation successful
0929 03:21:22.331903 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":77,"threads_started":1}
I20260929 03:21:22.332259 6294 raft_consensus.cc:504] T 00000000000000000000000000000000 P 322d1455dc2247a2ad27f51d40ec47a9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "322d1455dc2247a2ad27f51d40ec47a9" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 44805 } }
I20260929 03:21:22.332608 6294 leader_election.cc:302] T 00000000000000000000000000000000 P 322d1455dc2247a2ad27f51d40ec47a9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 322d1455dc2247a2ad27f51d40ec47a9; no voters:
I20260929 03:21:22.332825 6294 leader_election.cc:288] T 00000000000000000000000000000000 P 322d1455dc2247a2ad27f51d40ec47a9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:21:22.332852 6299 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 322d1455dc2247a2ad27f51d40ec47a9 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:21:22.333016 6299 raft_consensus.cc:686] T 00000000000000000000000000000000 P 322d1455dc2247a2ad27f51d40ec47a9 [term 1 LEADER]: Becoming Leader. State: Replica: 322d1455dc2247a2ad27f51d40ec47a9, State: Running, Role: LEADER
I20260929 03:21:22.333073 6294 sys_catalog.cc:564] T 00000000000000000000000000000000 P 322d1455dc2247a2ad27f51d40ec47a9 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:21:22.333097 6299 consensus_queue.cc:237] T 00000000000000000000000000000000 P 322d1455dc2247a2ad27f51d40ec47a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "322d1455dc2247a2ad27f51d40ec47a9" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 44805 } }
I20260929 03:21:22.333470 6300 sys_catalog.cc:455] T 00000000000000000000000000000000 P 322d1455dc2247a2ad27f51d40ec47a9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "322d1455dc2247a2ad27f51d40ec47a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "322d1455dc2247a2ad27f51d40ec47a9" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 44805 } } }
I20260929 03:21:22.333551 6300 sys_catalog.cc:458] T 00000000000000000000000000000000 P 322d1455dc2247a2ad27f51d40ec47a9 [sys.catalog]: This master's current role is: LEADER
I20260929 03:21:22.333513 6301 sys_catalog.cc:455] T 00000000000000000000000000000000 P 322d1455dc2247a2ad27f51d40ec47a9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 322d1455dc2247a2ad27f51d40ec47a9. Latest consensus state: current_term: 1 leader_uuid: "322d1455dc2247a2ad27f51d40ec47a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "322d1455dc2247a2ad27f51d40ec47a9" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 44805 } } }
I20260929 03:21:22.333666 6301 sys_catalog.cc:458] T 00000000000000000000000000000000 P 322d1455dc2247a2ad27f51d40ec47a9 [sys.catalog]: This master's current role is: LEADER
I20260929 03:21:22.333907 6306 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:21:22.334389 6306 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:21:22.335845 6306 catalog_manager.cc:1347] Generated new cluster ID: 0883a4d9efb64ab8a77f3107285a2cc4
I20260929 03:21:22.335902 6306 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:21:22.364476 6306 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:21:22.365254 6306 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:21:22.375283 6306 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 322d1455dc2247a2ad27f51d40ec47a9: Generated new TSK 0
I20260929 03:21:22.375490 6306 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 03:21:22.378139 21193 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuoaHbX/build/release/bin/kudu
/tmp/dist-test-taskuoaHbX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790651852476208-21193-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790651852476208-21193-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790651852476208-21193-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790651852476208-21193-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.178.65:0
--local_ip_for_outbound_sockets=127.20.178.65
--webserver_interface=127.20.178.65
--webserver_port=0
--tserver_master_addrs=127.20.178.126:44805
--builtin_ntp_servers=127.20.178.84:43541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 03:21:22.474454 6318 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:21:22.474690 6318 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:21:22.474736 6318 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:21:22.475925 6318 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:21:22.475989 6318 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:21:22.476033 6318 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.178.65
I20260929 03:21:22.477398 6318 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.178.84:43541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790651852476208-21193-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790651852476208-21193-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.178.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790651852476208-21193-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.178.65
--webserver_port=0
--tserver_master_addrs=127.20.178.126:44805
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6318
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.178.65
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790651852476208-21193-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:21:22.477659 6318 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:21:22.477990 6318 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:21:22.483587 6323 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:21:22.483661 6318 server_base.cc:1034] running on GCE node
W20260929 03:21:22.483608 6326 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:21:22.483779 6324 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:21:22.484086 6318 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:21:22.484335 6318 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:21:22.485493 6318 hybrid_clock.cc:648] HybridClock initialized: now 1790652082485468 us; error 41 us; skew 500 ppm
I20260929 03:21:22.486840 6318 webserver.cc:460] Webserver started at http://127.20.178.65:42813/ using document root <none> and password file <none>
I20260929 03:21:22.487088 6318 fs_manager.cc:359] Metadata directory not provided
I20260929 03:21:22.487133 6318 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:21:22.487284 6318 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:21:22.488327 6318 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790651852476208-21193-0/minicluster-data/ts-0/data/instance:
uuid: "20f1d5e194234d4a85252e2e64cfaf36"
format_stamp: "Formatted at 2026-09-29 03:21:22 on dist-test-slave-c9qh"
server_key: "8db81683c0b01caf03887e4eb2807000"
server_key_iv: "25752c94efd2421a4d85fab4c79390d4"
server_key_version: "encryptionkey@0"
I20260929 03:21:22.488744 6318 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790651852476208-21193-0/minicluster-data/ts-0/wal/instance:
uuid: "20f1d5e194234d4a85252e2e64cfaf36"
format_stamp: "Formatted at 2026-09-29 03:21:22 on dist-test-slave-c9qh"
server_key: "8db81683c0b01caf03887e4eb2807000"
server_key_iv: "25752c94efd2421a4d85fab4c79390d4"
server_key_version: "encryptionkey@0"
I20260929 03:21:22.490007 6318 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260929 03:21:22.490670 6332 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:21:22.490840 6318 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 03:21:22.490924 6318 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790651852476208-21193-0/minicluster-data/ts-0/data,/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790651852476208-21193-0/minicluster-data/ts-0/wal
uuid: "20f1d5e194234d4a85252e2e64cfaf36"
format_stamp: "Formatted at 2026-09-29 03:21:22 on dist-test-slave-c9qh"
server_key: "8db81683c0b01caf03887e4eb2807000"
server_key_iv: "25752c94efd2421a4d85fab4c79390d4"
server_key_version: "encryptionkey@0"
I20260929 03:21:22.490975 6318 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790651852476208-21193-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790651852476208-21193-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790651852476208-21193-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:21:22.510201 6318 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:21:22.510465 6318 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:21:22.510565 6318 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:21:22.510739 6318 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:21:22.511096 6318 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:21:22.511157 6318 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:21:22.511231 6318 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:21:22.511261 6318 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:21:22.516850 6318 rpc_server.cc:307] RPC server started. Bound to: 127.20.178.65:40777
I20260929 03:21:22.516927 6445 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.178.65:40777 every 8 connection(s)
I20260929 03:21:22.517256 6318 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790651852476208-21193-0/minicluster-data/ts-0/data/info.pb
I20260929 03:21:22.522382 6297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:22.518003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.65:44299 (local address 127.20.178.126:44805)
0929 03:21:22.518074 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0929 03:21:22.518075 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 03:21:22.518311 (+ 236us) server_negotiation.cc:407] Connection header received
0929 03:21:22.518796 (+ 485us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:21:22.518796 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:21:22.518806 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:21:22.518868 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:21:22.519538 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:22.520035 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:22.521108 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:22.521310 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:22.521374 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:21:22.522146 (+ 772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:21:22.522147 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:21:22.522147 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:21:22.522151 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:21:22.522165 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:21:22.522169 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:21:22.522219 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0929 03:21:22.522299 (+ 80us) server_negotiation.cc:299] Negotiation successful
0929 03:21:22.522323 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 03:21:22.522589 6448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:22.517986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:44805 (local address 127.20.178.65:44299)
0929 03:21:22.518178 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0929 03:21:22.518219 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 03:21:22.518706 (+ 487us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:21:22.518943 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:21:22.518947 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:21:22.518955 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:21:22.519435 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:21:22.519438 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:21:22.520123 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:21:22.520123 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:21:22.521020 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:21:22.521023 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:21:22.521390 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:21:22.521391 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:21:22.521492 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:21:22.522001 (+ 509us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 03:21:22.522001 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:21:22.522028 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 03:21:22.522051 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:21:22.522224 (+ 173us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:21:22.522228 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:21:22.522230 (+ 2us) client_negotiation.cc:769] Sending connection context
0929 03:21:22.522283 (+ 53us) client_negotiation.cc:240] Negotiation successful
0929 03:21:22.522357 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":98,"threads_started":1}
I20260929 03:21:22.523098 6446 heartbeater.cc:347] Connected to a master server at 127.20.178.126:44805
I20260929 03:21:22.523191 6446 heartbeater.cc:464] Registering TS with master...
I20260929 03:21:22.523439 6446 heartbeater.cc:511] Master 127.20.178.126:44805 requested a full tablet report, sending...
I20260929 03:21:22.523774 21193 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuoaHbX/build/release/bin/kudu as pid 6318
I20260929 03:21:22.523872 21193 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790651852476208-21193-0/minicluster-data/ts-0/wal/instance
I20260929 03:21:22.524051 21193 external_mini_cluster.cc:1428] Setting key a7923ca9ea9a368529a2546498aa5a2a
I20260929 03:21:22.524055 6250 ts_manager.cc:194] Registered new tserver with Master: 20f1d5e194234d4a85252e2e64cfaf36 (127.20.178.65:40777)
I20260929 03:21:22.524778 6250 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.178.65:44299
I20260929 03:21:22.525301 21193 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuoaHbX/build/release/bin/kudu
/tmp/dist-test-taskuoaHbX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790651852476208-21193-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790651852476208-21193-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790651852476208-21193-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790651852476208-21193-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.178.66:0
--local_ip_for_outbound_sockets=127.20.178.66
--webserver_interface=127.20.178.66
--webserver_port=0
--tserver_master_addrs=127.20.178.126:44805
--builtin_ntp_servers=127.20.178.84:43541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 03:21:22.605147 6449 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:21:22.605353 6449 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:21:22.605374 6449 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:21:22.606508 6449 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:21:22.606547 6449 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:21:22.606572 6449 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.178.66
I20260929 03:21:22.607908 6449 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.178.84:43541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790651852476208-21193-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790651852476208-21193-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.178.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790651852476208-21193-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.178.66
--webserver_port=0
--tserver_master_addrs=127.20.178.126:44805
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6449
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.178.66
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790651852476208-21193-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:21:22.608141 6449 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:21:22.608429 6449 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:21:22.613431 6454 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:21:22.613464 6457 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:21:22.613413 6455 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:21:22.613901 6449 server_base.cc:1034] running on GCE node
I20260929 03:21:22.614133 6449 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:21:22.614383 6449 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:21:22.615530 6449 hybrid_clock.cc:648] HybridClock initialized: now 1790652082615516 us; error 28 us; skew 500 ppm
I20260929 03:21:22.617233 6449 webserver.cc:460] Webserver started at http://127.20.178.66:40863/ using document root <none> and password file <none>
I20260929 03:21:22.617441 6449 fs_manager.cc:359] Metadata directory not provided
I20260929 03:21:22.617484 6449 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:21:22.617573 6449 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:21:22.618551 6449 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790651852476208-21193-0/minicluster-data/ts-1/data/instance:
uuid: "fc6fa9fee10f418eaafa8c9c58312122"
format_stamp: "Formatted at 2026-09-29 03:21:22 on dist-test-slave-c9qh"
server_key: "847ed3ee443e89cd7721f22e83a4b71c"
server_key_iv: "3400938af082ce4eb2986cb33da5f93a"
server_key_version: "encryptionkey@0"
I20260929 03:21:22.618861 6449 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790651852476208-21193-0/minicluster-data/ts-1/wal/instance:
uuid: "fc6fa9fee10f418eaafa8c9c58312122"
format_stamp: "Formatted at 2026-09-29 03:21:22 on dist-test-slave-c9qh"
server_key: "847ed3ee443e89cd7721f22e83a4b71c"
server_key_iv: "3400938af082ce4eb2986cb33da5f93a"
server_key_version: "encryptionkey@0"
I20260929 03:21:22.620133 6449 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 03:21:22.620836 6463 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:21:22.621011 6449 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260929 03:21:22.621109 6449 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790651852476208-21193-0/minicluster-data/ts-1/data,/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790651852476208-21193-0/minicluster-data/ts-1/wal
uuid: "fc6fa9fee10f418eaafa8c9c58312122"
format_stamp: "Formatted at 2026-09-29 03:21:22 on dist-test-slave-c9qh"
server_key: "847ed3ee443e89cd7721f22e83a4b71c"
server_key_iv: "3400938af082ce4eb2986cb33da5f93a"
server_key_version: "encryptionkey@0"
I20260929 03:21:22.621196 6449 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790651852476208-21193-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790651852476208-21193-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790651852476208-21193-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:21:22.631731 6449 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:21:22.632042 6449 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:21:22.632207 6449 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:21:22.632485 6449 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:21:22.632874 6449 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:21:22.632931 6449 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:21:22.632970 6449 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:21:22.632997 6449 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:21:22.639050 6449 rpc_server.cc:307] RPC server started. Bound to: 127.20.178.66:44491
I20260929 03:21:22.639086 6576 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.178.66:44491 every 8 connection(s)
I20260929 03:21:22.639482 6449 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790651852476208-21193-0/minicluster-data/ts-1/data/info.pb
I20260929 03:21:22.639654 21193 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuoaHbX/build/release/bin/kudu as pid 6449
I20260929 03:21:22.639758 21193 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790651852476208-21193-0/minicluster-data/ts-1/wal/instance
I20260929 03:21:22.639992 21193 external_mini_cluster.cc:1428] Setting key ae54f9c46e14a3e75d0bd804a98e9d36
I20260929 03:21:22.640987 21193 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuoaHbX/build/release/bin/kudu
/tmp/dist-test-taskuoaHbX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790651852476208-21193-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790651852476208-21193-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790651852476208-21193-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790651852476208-21193-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.178.67:0
--local_ip_for_outbound_sockets=127.20.178.67
--webserver_interface=127.20.178.67
--webserver_port=0
--tserver_master_addrs=127.20.178.126:44805
--builtin_ntp_servers=127.20.178.84:43541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 03:21:22.651407 6297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:22.640329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.66:53951 (local address 127.20.178.126:44805)
0929 03:21:22.640392 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0929 03:21:22.640395 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:21:22.640625 (+ 230us) server_negotiation.cc:407] Connection header received
0929 03:21:22.641302 (+ 677us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:21:22.641304 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:21:22.641316 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:21:22.641387 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:21:22.642025 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:22.642623 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:22.643621 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:22.643837 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:22.643922 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:21:22.651077 (+ 7155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:21:22.651078 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:21:22.651079 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:21:22.651084 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:21:22.651108 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:21:22.651112 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:21:22.651186 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0929 03:21:22.651291 (+ 105us) server_negotiation.cc:299] Negotiation successful
0929 03:21:22.651325 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260929 03:21:22.651641 6579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:22.640318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:44805 (local address 127.20.178.66:53951)
0929 03:21:22.640511 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0929 03:21:22.640553 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 03:21:22.641190 (+ 637us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:21:22.641441 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:21:22.641445 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:21:22.641453 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:21:22.641937 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:21:22.641942 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:21:22.642728 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:21:22.642729 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:21:22.643524 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:21:22.643526 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:21:22.643941 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:21:22.643942 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:21:22.644076 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:21:22.650908 (+ 6832us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 03:21:22.650909 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:21:22.650939 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 03:21:22.650963 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:21:22.651190 (+ 227us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:21:22.651198 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:21:22.651201 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 03:21:22.651293 (+ 92us) client_negotiation.cc:240] Negotiation successful
0929 03:21:22.651403 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":86,"threads_started":1}
I20260929 03:21:22.652153 6577 heartbeater.cc:347] Connected to a master server at 127.20.178.126:44805
I20260929 03:21:22.652295 6577 heartbeater.cc:464] Registering TS with master...
I20260929 03:21:22.652493 6577 heartbeater.cc:511] Master 127.20.178.126:44805 requested a full tablet report, sending...
I20260929 03:21:22.652894 6250 ts_manager.cc:194] Registered new tserver with Master: fc6fa9fee10f418eaafa8c9c58312122 (127.20.178.66:44491)
I20260929 03:21:22.653282 6250 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.178.66:53951
W20260929 03:21:22.722373 6580 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:21:22.722599 6580 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:21:22.722633 6580 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:21:22.723836 6580 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:21:22.723898 6580 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:21:22.723942 6580 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.178.67
I20260929 03:21:22.725492 6580 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.178.84:43541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790651852476208-21193-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790651852476208-21193-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.178.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790651852476208-21193-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.178.67
--webserver_port=0
--tserver_master_addrs=127.20.178.126:44805
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6580
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.178.67
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790651852476208-21193-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:21:22.725751 6580 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:21:22.726049 6580 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:21:22.731084 6585 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:21:22.731076 6586 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:21:22.731120 6580 server_base.cc:1034] running on GCE node
W20260929 03:21:22.731073 6588 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:21:22.731539 6580 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:21:22.731789 6580 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:21:22.732975 6580 hybrid_clock.cc:648] HybridClock initialized: now 1790652082732957 us; error 27 us; skew 500 ppm
I20260929 03:21:22.734395 6580 webserver.cc:460] Webserver started at http://127.20.178.67:42689/ using document root <none> and password file <none>
I20260929 03:21:22.734642 6580 fs_manager.cc:359] Metadata directory not provided
I20260929 03:21:22.734709 6580 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:21:22.734858 6580 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:21:22.735980 6580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790651852476208-21193-0/minicluster-data/ts-2/data/instance:
uuid: "4d156675a70d485eae55a7f5a628a932"
format_stamp: "Formatted at 2026-09-29 03:21:22 on dist-test-slave-c9qh"
server_key: "3df3c7be7d2af8bf673f2c6e33211a37"
server_key_iv: "b155342108326f1d8e736022211e9dc3"
server_key_version: "encryptionkey@0"
I20260929 03:21:22.736354 6580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790651852476208-21193-0/minicluster-data/ts-2/wal/instance:
uuid: "4d156675a70d485eae55a7f5a628a932"
format_stamp: "Formatted at 2026-09-29 03:21:22 on dist-test-slave-c9qh"
server_key: "3df3c7be7d2af8bf673f2c6e33211a37"
server_key_iv: "b155342108326f1d8e736022211e9dc3"
server_key_version: "encryptionkey@0"
I20260929 03:21:22.737804 6580 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 03:21:22.738670 6594 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:21:22.738854 6580 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 03:21:22.738952 6580 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790651852476208-21193-0/minicluster-data/ts-2/data,/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790651852476208-21193-0/minicluster-data/ts-2/wal
uuid: "4d156675a70d485eae55a7f5a628a932"
format_stamp: "Formatted at 2026-09-29 03:21:22 on dist-test-slave-c9qh"
server_key: "3df3c7be7d2af8bf673f2c6e33211a37"
server_key_iv: "b155342108326f1d8e736022211e9dc3"
server_key_version: "encryptionkey@0"
I20260929 03:21:22.739037 6580 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790651852476208-21193-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790651852476208-21193-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790651852476208-21193-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:21:22.752250 6580 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:21:22.752545 6580 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:21:22.752727 6580 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:21:22.752943 6580 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:21:22.753278 6580 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:21:22.753338 6580 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:21:22.753381 6580 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:21:22.753410 6580 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:21:22.759155 6580 rpc_server.cc:307] RPC server started. Bound to: 127.20.178.67:45241
I20260929 03:21:22.759197 6707 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.178.67:45241 every 8 connection(s)
I20260929 03:21:22.759523 6580 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790651852476208-21193-0/minicluster-data/ts-2/data/info.pb
I20260929 03:21:22.764417 6297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:22.760203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.67:40019 (local address 127.20.178.126:44805)
0929 03:21:22.760261 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0929 03:21:22.760263 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:21:22.760562 (+ 299us) server_negotiation.cc:407] Connection header received
0929 03:21:22.761088 (+ 526us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:21:22.761089 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:21:22.761099 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:21:22.761153 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:21:22.761800 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:22.762302 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:22.763212 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:22.763391 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:22.763439 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:21:22.764188 (+ 749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:21:22.764189 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:21:22.764189 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:21:22.764193 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:21:22.764208 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:21:22.764211 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:21:22.764278 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0929 03:21:22.764339 (+ 61us) server_negotiation.cc:299] Negotiation successful
0929 03:21:22.764359 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260929 03:21:22.764606 6710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:22.760212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:44805 (local address 127.20.178.67:40019)
0929 03:21:22.760432 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0929 03:21:22.760465 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0929 03:21:22.760996 (+ 531us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:21:22.761210 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:21:22.761213 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:21:22.761222 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:21:22.761717 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:21:22.761722 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:21:22.762385 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:21:22.762385 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:21:22.763137 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:21:22.763139 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:21:22.763463 (+ 324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:21:22.763464 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:21:22.763570 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:21:22.764038 (+ 468us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 03:21:22.764038 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:21:22.764085 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 03:21:22.764107 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:21:22.764278 (+ 171us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:21:22.764280 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:21:22.764283 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 03:21:22.764334 (+ 51us) client_negotiation.cc:240] Negotiation successful
0929 03:21:22.764407 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":131,"threads_started":1}
I20260929 03:21:22.765133 6708 heartbeater.cc:347] Connected to a master server at 127.20.178.126:44805
I20260929 03:21:22.765282 6708 heartbeater.cc:464] Registering TS with master...
I20260929 03:21:22.765494 6708 heartbeater.cc:511] Master 127.20.178.126:44805 requested a full tablet report, sending...
I20260929 03:21:22.765690 21193 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuoaHbX/build/release/bin/kudu as pid 6580
I20260929 03:21:22.765790 21193 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790651852476208-21193-0/minicluster-data/ts-2/wal/instance
I20260929 03:21:22.765925 6250 ts_manager.cc:194] Registered new tserver with Master: 4d156675a70d485eae55a7f5a628a932 (127.20.178.67:45241)
I20260929 03:21:22.766004 21193 external_mini_cluster.cc:1428] Setting key 17d9ed945700d2954d150644190b301d
I20260929 03:21:22.766391 6250 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.178.67:40019
I20260929 03:21:22.766952 21193 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 03:21:22.770819 6297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:22.768259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56716 (local address 127.20.178.126:44805)
0929 03:21:22.768296 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0929 03:21:22.768298 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:21:22.768429 (+ 131us) server_negotiation.cc:407] Connection header received
0929 03:21:22.768476 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:21:22.768477 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:21:22.768485 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:21:22.768525 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:21:22.768680 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:22.769132 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:22.769871 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:22.770039 (+ 168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:22.770090 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:21:22.770643 (+ 553us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:21:22.770644 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:21:22.770644 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:21:22.770648 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:21:22.770659 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:21:22.770663 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:21:22.770708 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0929 03:21:22.770751 (+ 43us) server_negotiation.cc:299] Negotiation successful
0929 03:21:22.770768 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260929 03:21:22.772264 6250 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56716:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 03:21:22.773804 6250 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 03:21:22.779887 6717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:22.776021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.66:44491 (local address 127.0.0.1:58594)
0929 03:21:22.776197 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0929 03:21:22.776238 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 03:21:22.776298 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:21:22.776622 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:21:22.776625 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:21:22.776628 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:21:22.776718 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:21:22.776720 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:21:22.777654 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:21:22.777655 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:21:22.778778 (+ 1123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:21:22.778781 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:21:22.779598 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:21:22.779599 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:21:22.779672 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:21:22.779684 (+ 12us) client_negotiation.cc:769] Sending connection context
0929 03:21:22.779761 (+ 77us) client_negotiation.cc:240] Negotiation successful
0929 03:21:22.779811 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":66,"threads_started":1}
I20260929 03:21:22.779902 6719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:22.776201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58594 (local address 127.20.178.66:44491)
0929 03:21:22.776347 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0929 03:21:22.776353 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:21:22.776364 (+ 11us) server_negotiation.cc:407] Connection header received
0929 03:21:22.776406 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:21:22.776408 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:21:22.776425 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:21:22.776537 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:21:22.776803 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:22.777564 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:22.778888 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:22.779508 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:22.779598 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:21:22.779690 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0929 03:21:22.779775 (+ 85us) server_negotiation.cc:299] Negotiation successful
0929 03:21:22.779811 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98,"thread_start_us":68,"threads_started":1}
I20260929 03:21:22.780194 6720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:22.776279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.65:40777 (local address 127.0.0.1:47264)
0929 03:21:22.776492 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0929 03:21:22.776502 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 03:21:22.776543 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:21:22.776903 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:21:22.776904 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:21:22.776905 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:21:22.776937 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:21:22.776938 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:21:22.778206 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:21:22.778207 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:21:22.779058 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:21:22.779060 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:21:22.779975 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:21:22.779977 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:21:22.780038 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:21:22.780043 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 03:21:22.780073 (+ 30us) client_negotiation.cc:240] Negotiation successful
0929 03:21:22.780110 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":70,"threads_started":1}
I20260929 03:21:22.780339 6718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:22.776107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39060 (local address 127.20.178.67:45241)
0929 03:21:22.776372 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0929 03:21:22.776377 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:21:22.776526 (+ 149us) server_negotiation.cc:407] Connection header received
0929 03:21:22.776569 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:21:22.776571 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:21:22.776588 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:21:22.776678 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:21:22.776896 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:22.778008 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:22.779214 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:22.779874 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:22.779943 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:21:22.780020 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0929 03:21:22.780191 (+ 171us) server_negotiation.cc:299] Negotiation successful
0929 03:21:22.780230 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":232,"thread_start_us":99,"threads_started":1}
I20260929 03:21:22.780400 6716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:22.776021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.67:45241 (local address 127.0.0.1:39060)
0929 03:21:22.776408 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0929 03:21:22.776417 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 03:21:22.776460 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:21:22.776763 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:21:22.776764 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:21:22.776767 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:21:22.776812 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:21:22.776814 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:21:22.778106 (+ 1292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:21:22.778107 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:21:22.779147 (+ 1040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:21:22.779150 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:21:22.780049 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:21:22.780050 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:21:22.780103 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:21:22.780105 (+ 2us) client_negotiation.cc:769] Sending connection context
0929 03:21:22.780160 (+ 55us) client_negotiation.cc:240] Negotiation successful
0929 03:21:22.780190 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":377,"mutex_wait_us":2,"spinlock_wait_cycles":768,"thread_start_us":100,"threads_started":1}
I20260929 03:21:22.780505 6721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:22.776510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47264 (local address 127.20.178.65:40777)
0929 03:21:22.776743 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0929 03:21:22.776748 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:21:22.776756 (+ 8us) server_negotiation.cc:407] Connection header received
0929 03:21:22.776791 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:21:22.776794 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:21:22.776808 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:21:22.777035 (+ 227us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:21:22.777091 (+ 56us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:22.778130 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:22.779144 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:22.779870 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:22.780247 (+ 377us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:21:22.780347 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0929 03:21:22.780372 (+ 25us) server_negotiation.cc:299] Negotiation successful
0929 03:21:22.780408 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":194,"thread_start_us":86,"threads_started":1}
I20260929 03:21:22.781162 6642 tablet_service.cc:1467] Processing CreateTablet for tablet 11f1176973e04097b0efe932c9d1cc34 (DEFAULT_TABLE table=test-table [id=c1a978fcf4294ad9b01fca0e78400867]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:21:22.781339 6380 tablet_service.cc:1467] Processing CreateTablet for tablet 11f1176973e04097b0efe932c9d1cc34 (DEFAULT_TABLE table=test-table [id=c1a978fcf4294ad9b01fca0e78400867]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:21:22.781394 6511 tablet_service.cc:1467] Processing CreateTablet for tablet 11f1176973e04097b0efe932c9d1cc34 (DEFAULT_TABLE table=test-table [id=c1a978fcf4294ad9b01fca0e78400867]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:21:22.781595 6642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11f1176973e04097b0efe932c9d1cc34. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:21:22.781677 6380 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11f1176973e04097b0efe932c9d1cc34. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:21:22.781677 6511 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11f1176973e04097b0efe932c9d1cc34. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:21:22.784360 6722 tablet_bootstrap.cc:492] T 11f1176973e04097b0efe932c9d1cc34 P 4d156675a70d485eae55a7f5a628a932: Bootstrap starting.
I20260929 03:21:22.784922 6723 tablet_bootstrap.cc:492] T 11f1176973e04097b0efe932c9d1cc34 P 20f1d5e194234d4a85252e2e64cfaf36: Bootstrap starting.
I20260929 03:21:22.785075 6722 tablet_bootstrap.cc:654] T 11f1176973e04097b0efe932c9d1cc34 P 4d156675a70d485eae55a7f5a628a932: Neither blocks nor log segments found. Creating new log.
I20260929 03:21:22.785128 6724 tablet_bootstrap.cc:492] T 11f1176973e04097b0efe932c9d1cc34 P fc6fa9fee10f418eaafa8c9c58312122: Bootstrap starting.
I20260929 03:21:22.785410 6722 log.cc:824] T 11f1176973e04097b0efe932c9d1cc34 P 4d156675a70d485eae55a7f5a628a932: Log is configured to *not* fsync() on all Append() calls
I20260929 03:21:22.785633 6723 tablet_bootstrap.cc:654] T 11f1176973e04097b0efe932c9d1cc34 P 20f1d5e194234d4a85252e2e64cfaf36: Neither blocks nor log segments found. Creating new log.
I20260929 03:21:22.785763 6724 tablet_bootstrap.cc:654] T 11f1176973e04097b0efe932c9d1cc34 P fc6fa9fee10f418eaafa8c9c58312122: Neither blocks nor log segments found. Creating new log.
I20260929 03:21:22.785985 6723 log.cc:824] T 11f1176973e04097b0efe932c9d1cc34 P 20f1d5e194234d4a85252e2e64cfaf36: Log is configured to *not* fsync() on all Append() calls
I20260929 03:21:22.786043 6724 log.cc:824] T 11f1176973e04097b0efe932c9d1cc34 P fc6fa9fee10f418eaafa8c9c58312122: Log is configured to *not* fsync() on all Append() calls
I20260929 03:21:22.786289 6722 tablet_bootstrap.cc:492] T 11f1176973e04097b0efe932c9d1cc34 P 4d156675a70d485eae55a7f5a628a932: No bootstrap required, opened a new log
I20260929 03:21:22.786375 6722 ts_tablet_manager.cc:1397] T 11f1176973e04097b0efe932c9d1cc34 P 4d156675a70d485eae55a7f5a628a932: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260929 03:21:22.786720 6723 tablet_bootstrap.cc:492] T 11f1176973e04097b0efe932c9d1cc34 P 20f1d5e194234d4a85252e2e64cfaf36: No bootstrap required, opened a new log
I20260929 03:21:22.786755 6724 tablet_bootstrap.cc:492] T 11f1176973e04097b0efe932c9d1cc34 P fc6fa9fee10f418eaafa8c9c58312122: No bootstrap required, opened a new log
I20260929 03:21:22.786803 6723 ts_tablet_manager.cc:1397] T 11f1176973e04097b0efe932c9d1cc34 P 20f1d5e194234d4a85252e2e64cfaf36: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 03:21:22.786810 6724 ts_tablet_manager.cc:1397] T 11f1176973e04097b0efe932c9d1cc34 P fc6fa9fee10f418eaafa8c9c58312122: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 03:21:22.788542 6722 raft_consensus.cc:348] T 11f1176973e04097b0efe932c9d1cc34 P 4d156675a70d485eae55a7f5a628a932 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc6fa9fee10f418eaafa8c9c58312122" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 44491 } } peers { permanent_uuid: "4d156675a70d485eae55a7f5a628a932" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 45241 } } peers { permanent_uuid: "20f1d5e194234d4a85252e2e64cfaf36" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 40777 } }
I20260929 03:21:22.788758 6722 raft_consensus.cc:374] T 11f1176973e04097b0efe932c9d1cc34 P 4d156675a70d485eae55a7f5a628a932 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:21:22.788807 6722 raft_consensus.cc:729] T 11f1176973e04097b0efe932c9d1cc34 P 4d156675a70d485eae55a7f5a628a932 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d156675a70d485eae55a7f5a628a932, State: Initialized, Role: FOLLOWER
I20260929 03:21:22.788933 6722 consensus_queue.cc:260] T 11f1176973e04097b0efe932c9d1cc34 P 4d156675a70d485eae55a7f5a628a932 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc6fa9fee10f418eaafa8c9c58312122" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 44491 } } peers { permanent_uuid: "4d156675a70d485eae55a7f5a628a932" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 45241 } } peers { permanent_uuid: "20f1d5e194234d4a85252e2e64cfaf36" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 40777 } }
I20260929 03:21:22.789242 6708 heartbeater.cc:503] Master 127.20.178.126:44805 was elected leader, sending a full tablet report...
I20260929 03:21:22.789227 6722 ts_tablet_manager.cc:1428] T 11f1176973e04097b0efe932c9d1cc34 P 4d156675a70d485eae55a7f5a628a932: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260929 03:21:22.789408 6724 raft_consensus.cc:348] T 11f1176973e04097b0efe932c9d1cc34 P fc6fa9fee10f418eaafa8c9c58312122 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc6fa9fee10f418eaafa8c9c58312122" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 44491 } } peers { permanent_uuid: "4d156675a70d485eae55a7f5a628a932" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 45241 } } peers { permanent_uuid: "20f1d5e194234d4a85252e2e64cfaf36" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 40777 } }
I20260929 03:21:22.789552 6724 raft_consensus.cc:374] T 11f1176973e04097b0efe932c9d1cc34 P fc6fa9fee10f418eaafa8c9c58312122 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:21:22.789600 6724 raft_consensus.cc:729] T 11f1176973e04097b0efe932c9d1cc34 P fc6fa9fee10f418eaafa8c9c58312122 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc6fa9fee10f418eaafa8c9c58312122, State: Initialized, Role: FOLLOWER
I20260929 03:21:22.789408 6723 raft_consensus.cc:348] T 11f1176973e04097b0efe932c9d1cc34 P 20f1d5e194234d4a85252e2e64cfaf36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc6fa9fee10f418eaafa8c9c58312122" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 44491 } } peers { permanent_uuid: "4d156675a70d485eae55a7f5a628a932" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 45241 } } peers { permanent_uuid: "20f1d5e194234d4a85252e2e64cfaf36" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 40777 } }
I20260929 03:21:22.789639 6723 raft_consensus.cc:374] T 11f1176973e04097b0efe932c9d1cc34 P 20f1d5e194234d4a85252e2e64cfaf36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:21:22.789673 6723 raft_consensus.cc:729] T 11f1176973e04097b0efe932c9d1cc34 P 20f1d5e194234d4a85252e2e64cfaf36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20f1d5e194234d4a85252e2e64cfaf36, State: Initialized, Role: FOLLOWER
I20260929 03:21:22.789712 6724 consensus_queue.cc:260] T 11f1176973e04097b0efe932c9d1cc34 P fc6fa9fee10f418eaafa8c9c58312122 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc6fa9fee10f418eaafa8c9c58312122" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 44491 } } peers { permanent_uuid: "4d156675a70d485eae55a7f5a628a932" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 45241 } } peers { permanent_uuid: "20f1d5e194234d4a85252e2e64cfaf36" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 40777 } }
I20260929 03:21:22.789767 6723 consensus_queue.cc:260] T 11f1176973e04097b0efe932c9d1cc34 P 20f1d5e194234d4a85252e2e64cfaf36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc6fa9fee10f418eaafa8c9c58312122" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 44491 } } peers { permanent_uuid: "4d156675a70d485eae55a7f5a628a932" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 45241 } } peers { permanent_uuid: "20f1d5e194234d4a85252e2e64cfaf36" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 40777 } }
I20260929 03:21:22.789991 6724 ts_tablet_manager.cc:1428] T 11f1176973e04097b0efe932c9d1cc34 P fc6fa9fee10f418eaafa8c9c58312122: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260929 03:21:22.790041 6577 heartbeater.cc:503] Master 127.20.178.126:44805 was elected leader, sending a full tablet report...
I20260929 03:21:22.790271 6723 ts_tablet_manager.cc:1428] T 11f1176973e04097b0efe932c9d1cc34 P 20f1d5e194234d4a85252e2e64cfaf36: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260929 03:21:22.790396 6446 heartbeater.cc:503] Master 127.20.178.126:44805 was elected leader, sending a full tablet report...
I20260929 03:21:22.830263 6728 raft_consensus.cc:482] T 11f1176973e04097b0efe932c9d1cc34 P 4d156675a70d485eae55a7f5a628a932 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:21:22.830384 6728 raft_consensus.cc:504] T 11f1176973e04097b0efe932c9d1cc34 P 4d156675a70d485eae55a7f5a628a932 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc6fa9fee10f418eaafa8c9c58312122" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 44491 } } peers { permanent_uuid: "4d156675a70d485eae55a7f5a628a932" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 45241 } } peers { permanent_uuid: "20f1d5e194234d4a85252e2e64cfaf36" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 40777 } }
I20260929 03:21:22.830677 6728 leader_election.cc:288] T 11f1176973e04097b0efe932c9d1cc34 P 4d156675a70d485eae55a7f5a628a932 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fc6fa9fee10f418eaafa8c9c58312122 (127.20.178.66:44491), 20f1d5e194234d4a85252e2e64cfaf36 (127.20.178.65:40777)
I20260929 03:21:22.834178 6710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:22.830880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.66:44491 (local address 127.20.178.67:48245)
0929 03:21:22.830963 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0929 03:21:22.830976 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 03:21:22.831033 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:21:22.831299 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:21:22.831300 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:21:22.831302 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:21:22.831374 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:21:22.831375 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:21:22.832085 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:21:22.832086 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:21:22.833118 (+ 1032us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:21:22.833121 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:21:22.833895 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:21:22.833896 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:21:22.833998 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:21:22.834002 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 03:21:22.834071 (+ 69us) client_negotiation.cc:240] Negotiation successful
0929 03:21:22.834111 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55,"mutex_wait_us":16}
I20260929 03:21:22.834177 6719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:22.830976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.67:48245 (local address 127.20.178.66:44491)
0929 03:21:22.831042 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0929 03:21:22.831044 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:21:22.831054 (+ 10us) server_negotiation.cc:407] Connection header received
0929 03:21:22.831163 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:21:22.831164 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:21:22.831171 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:21:22.831292 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:21:22.831446 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:22.831990 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:22.833260 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:22.833819 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:22.833897 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:21:22.833906 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0929 03:21:22.834079 (+ 173us) server_negotiation.cc:299] Negotiation successful
0929 03:21:22.834120 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 03:21:22.834530 6731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:22.831021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.65:40777 (local address 127.20.178.67:51733)
0929 03:21:22.831153 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0929 03:21:22.831163 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 03:21:22.831251 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:21:22.831386 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:21:22.831386 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:21:22.831388 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:21:22.831457 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:21:22.831458 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:21:22.832108 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:21:22.832109 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:21:22.833133 (+ 1024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:21:22.833135 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:21:22.834298 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:21:22.834299 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:21:22.834371 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:21:22.834376 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 03:21:22.834427 (+ 51us) client_negotiation.cc:240] Negotiation successful
0929 03:21:22.834462 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":89,"threads_started":1}
I20260929 03:21:22.834554 6721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:22.831073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.67:51733 (local address 127.20.178.65:40777)
0929 03:21:22.831123 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0929 03:21:22.831126 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:21:22.831237 (+ 111us) server_negotiation.cc:407] Connection header received
0929 03:21:22.831306 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:21:22.831308 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:21:22.831319 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:21:22.831381 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:21:22.831534 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:22.832024 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:22.833261 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:22.833820 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:22.833898 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:21:22.833906 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0929 03:21:22.834438 (+ 532us) server_negotiation.cc:299] Negotiation successful
0929 03:21:22.834485 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260929 03:21:22.834556 6531 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11f1176973e04097b0efe932c9d1cc34" candidate_uuid: "4d156675a70d485eae55a7f5a628a932" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc6fa9fee10f418eaafa8c9c58312122" is_pre_election: true
I20260929 03:21:22.834725 6531 raft_consensus.cc:2454] T 11f1176973e04097b0efe932c9d1cc34 P fc6fa9fee10f418eaafa8c9c58312122 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d156675a70d485eae55a7f5a628a932 in term 0.
I20260929 03:21:22.834944 6598 leader_election.cc:302] T 11f1176973e04097b0efe932c9d1cc34 P 4d156675a70d485eae55a7f5a628a932 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d156675a70d485eae55a7f5a628a932, fc6fa9fee10f418eaafa8c9c58312122; no voters:
I20260929 03:21:22.834929 6400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11f1176973e04097b0efe932c9d1cc34" candidate_uuid: "4d156675a70d485eae55a7f5a628a932" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20f1d5e194234d4a85252e2e64cfaf36" is_pre_election: true
I20260929 03:21:22.835085 6400 raft_consensus.cc:2454] T 11f1176973e04097b0efe932c9d1cc34 P 20f1d5e194234d4a85252e2e64cfaf36 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d156675a70d485eae55a7f5a628a932 in term 0.
I20260929 03:21:22.835080 6728 raft_consensus.cc:2781] T 11f1176973e04097b0efe932c9d1cc34 P 4d156675a70d485eae55a7f5a628a932 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 03:21:22.835129 6728 raft_consensus.cc:482] T 11f1176973e04097b0efe932c9d1cc34 P 4d156675a70d485eae55a7f5a628a932 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 03:21:22.835152 6728 raft_consensus.cc:3037] T 11f1176973e04097b0efe932c9d1cc34 P 4d156675a70d485eae55a7f5a628a932 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:21:22.835878 6728 raft_consensus.cc:504] T 11f1176973e04097b0efe932c9d1cc34 P 4d156675a70d485eae55a7f5a628a932 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc6fa9fee10f418eaafa8c9c58312122" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 44491 } } peers { permanent_uuid: "4d156675a70d485eae55a7f5a628a932" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 45241 } } peers { permanent_uuid: "20f1d5e194234d4a85252e2e64cfaf36" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 40777 } }
I20260929 03:21:22.836021 6728 leader_election.cc:288] T 11f1176973e04097b0efe932c9d1cc34 P 4d156675a70d485eae55a7f5a628a932 [CANDIDATE]: Term 1 election: Requested vote from peers fc6fa9fee10f418eaafa8c9c58312122 (127.20.178.66:44491), 20f1d5e194234d4a85252e2e64cfaf36 (127.20.178.65:40777)
I20260929 03:21:22.836171 6400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11f1176973e04097b0efe932c9d1cc34" candidate_uuid: "4d156675a70d485eae55a7f5a628a932" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20f1d5e194234d4a85252e2e64cfaf36"
I20260929 03:21:22.836187 6531 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11f1176973e04097b0efe932c9d1cc34" candidate_uuid: "4d156675a70d485eae55a7f5a628a932" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc6fa9fee10f418eaafa8c9c58312122"
I20260929 03:21:22.836246 6400 raft_consensus.cc:3037] T 11f1176973e04097b0efe932c9d1cc34 P 20f1d5e194234d4a85252e2e64cfaf36 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:21:22.836278 6531 raft_consensus.cc:3037] T 11f1176973e04097b0efe932c9d1cc34 P fc6fa9fee10f418eaafa8c9c58312122 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:21:22.837039 6400 raft_consensus.cc:2454] T 11f1176973e04097b0efe932c9d1cc34 P 20f1d5e194234d4a85252e2e64cfaf36 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d156675a70d485eae55a7f5a628a932 in term 1.
I20260929 03:21:22.837141 6531 raft_consensus.cc:2454] T 11f1176973e04097b0efe932c9d1cc34 P fc6fa9fee10f418eaafa8c9c58312122 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d156675a70d485eae55a7f5a628a932 in term 1.
I20260929 03:21:22.837211 6598 leader_election.cc:302] T 11f1176973e04097b0efe932c9d1cc34 P 4d156675a70d485eae55a7f5a628a932 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20f1d5e194234d4a85252e2e64cfaf36, 4d156675a70d485eae55a7f5a628a932; no voters:
I20260929 03:21:22.837342 6728 raft_consensus.cc:2781] T 11f1176973e04097b0efe932c9d1cc34 P 4d156675a70d485eae55a7f5a628a932 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:21:22.837520 6728 raft_consensus.cc:686] T 11f1176973e04097b0efe932c9d1cc34 P 4d156675a70d485eae55a7f5a628a932 [term 1 LEADER]: Becoming Leader. State: Replica: 4d156675a70d485eae55a7f5a628a932, State: Running, Role: LEADER
I20260929 03:21:22.837622 6728 consensus_queue.cc:237] T 11f1176973e04097b0efe932c9d1cc34 P 4d156675a70d485eae55a7f5a628a932 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc6fa9fee10f418eaafa8c9c58312122" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 44491 } } peers { permanent_uuid: "4d156675a70d485eae55a7f5a628a932" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 45241 } } peers { permanent_uuid: "20f1d5e194234d4a85252e2e64cfaf36" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 40777 } }
I20260929 03:21:22.838497 6258 catalog_manager.cc:5495] T 11f1176973e04097b0efe932c9d1cc34 P 4d156675a70d485eae55a7f5a628a932 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4d156675a70d485eae55a7f5a628a932 (127.20.178.67). New cstate: current_term: 1 leader_uuid: "4d156675a70d485eae55a7f5a628a932" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc6fa9fee10f418eaafa8c9c58312122" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 44491 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d156675a70d485eae55a7f5a628a932" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 45241 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "20f1d5e194234d4a85252e2e64cfaf36" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 40777 } health_report { overall_health: UNKNOWN } } }
I20260929 03:21:22.852422 6718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:22.850058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39074 (local address 127.20.178.67:45241)
0929 03:21:22.850117 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0929 03:21:22.850119 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:21:22.850128 (+ 9us) server_negotiation.cc:407] Connection header received
0929 03:21:22.850169 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:21:22.850170 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:21:22.850185 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:21:22.850257 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:21:22.850354 (+ 97us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:22.850801 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:22.851613 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:22.851799 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:22.851857 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:21:22.851986 (+ 129us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:21:22.852079 (+ 93us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:21:22.852141 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0929 03:21:22.852251 (+ 110us) server_negotiation.cc:299] Negotiation successful
0929 03:21:22.852288 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260929 03:21:22.854292 6531 raft_consensus.cc:1261] T 11f1176973e04097b0efe932c9d1cc34 P fc6fa9fee10f418eaafa8c9c58312122 [term 1 FOLLOWER]: Refusing update from remote peer 4d156675a70d485eae55a7f5a628a932: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:21:22.854285 6400 raft_consensus.cc:1261] T 11f1176973e04097b0efe932c9d1cc34 P 20f1d5e194234d4a85252e2e64cfaf36 [term 1 FOLLOWER]: Refusing update from remote peer 4d156675a70d485eae55a7f5a628a932: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:21:22.854691 6728 consensus_queue.cc:1035] T 11f1176973e04097b0efe932c9d1cc34 P 4d156675a70d485eae55a7f5a628a932 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc6fa9fee10f418eaafa8c9c58312122" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 44491 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:21:22.854856 6732 consensus_queue.cc:1035] T 11f1176973e04097b0efe932c9d1cc34 P 4d156675a70d485eae55a7f5a628a932 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20f1d5e194234d4a85252e2e64cfaf36" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 40777 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:21:22.857679 6735 mvcc.cc:205] Tried to move back new op lower bound from 7334510931368275968 to 7334510931303739392. Current Snapshot: MvccSnapshot[applied={T|T < 7334510931368275968}]
I20260929 03:21:22.857877 6736 mvcc.cc:205] Tried to move back new op lower bound from 7334510931368275968 to 7334510931303739392. Current Snapshot: MvccSnapshot[applied={T|T < 7334510931368275968}]
I20260929 03:21:22.860162 6257 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56716:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:21:22.860270 6257 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56716:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:21:22.861050 6257 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 322d1455dc2247a2ad27f51d40ec47a9: Sending DeleteTablet for 3 replicas of tablet 11f1176973e04097b0efe932c9d1cc34
I20260929 03:21:22.861398 6511 tablet_service.cc:1514] Processing DeleteTablet for tablet 11f1176973e04097b0efe932c9d1cc34 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:21:22 UTC) from {username='slave'} at 127.0.0.1:58594
I20260929 03:21:22.861449 6380 tablet_service.cc:1514] Processing DeleteTablet for tablet 11f1176973e04097b0efe932c9d1cc34 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:21:22 UTC) from {username='slave'} at 127.0.0.1:47264
I20260929 03:21:22.861665 6642 tablet_service.cc:1514] Processing DeleteTablet for tablet 11f1176973e04097b0efe932c9d1cc34 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:21:22 UTC) from {username='slave'} at 127.0.0.1:39060
I20260929 03:21:22.861922 6747 tablet_replica.cc:330] T 11f1176973e04097b0efe932c9d1cc34 P 4d156675a70d485eae55a7f5a628a932: stopping tablet replica
I20260929 03:21:22.861994 21193 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuoaHbX/build/release/bin/kudu with pid 6318
I20260929 03:21:22.864136 6745 tablet_replica.cc:330] T 11f1176973e04097b0efe932c9d1cc34 P fc6fa9fee10f418eaafa8c9c58312122: stopping tablet replica
I20260929 03:21:22.864214 6747 raft_consensus.cc:2229] T 11f1176973e04097b0efe932c9d1cc34 P 4d156675a70d485eae55a7f5a628a932 [term 1 LEADER]: Raft consensus shutting down.
I20260929 03:21:22.864256 6745 raft_consensus.cc:2229] T 11f1176973e04097b0efe932c9d1cc34 P fc6fa9fee10f418eaafa8c9c58312122 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:21:22.864337 6747 raft_consensus.cc:2258] T 11f1176973e04097b0efe932c9d1cc34 P 4d156675a70d485eae55a7f5a628a932 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:21:22.864351 6745 raft_consensus.cc:2258] T 11f1176973e04097b0efe932c9d1cc34 P fc6fa9fee10f418eaafa8c9c58312122 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:21:22.864799 6747 ts_tablet_manager.cc:1905] T 11f1176973e04097b0efe932c9d1cc34 P 4d156675a70d485eae55a7f5a628a932: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:21:22.864799 6745 ts_tablet_manager.cc:1905] T 11f1176973e04097b0efe932c9d1cc34 P fc6fa9fee10f418eaafa8c9c58312122: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:21:22.866667 6747 ts_tablet_manager.cc:1918] T 11f1176973e04097b0efe932c9d1cc34 P 4d156675a70d485eae55a7f5a628a932: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:21:22.866667 6745 ts_tablet_manager.cc:1918] T 11f1176973e04097b0efe932c9d1cc34 P fc6fa9fee10f418eaafa8c9c58312122: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:21:22.866751 6747 log.cc:1196] T 11f1176973e04097b0efe932c9d1cc34 P 4d156675a70d485eae55a7f5a628a932: Deleting WAL directory at /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790651852476208-21193-0/minicluster-data/ts-2/wal/wals/11f1176973e04097b0efe932c9d1cc34
I20260929 03:21:22.866751 6745 log.cc:1196] T 11f1176973e04097b0efe932c9d1cc34 P fc6fa9fee10f418eaafa8c9c58312122: Deleting WAL directory at /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790651852476208-21193-0/minicluster-data/ts-1/wal/wals/11f1176973e04097b0efe932c9d1cc34
I20260929 03:21:22.867009 6745 ts_tablet_manager.cc:1939] T 11f1176973e04097b0efe932c9d1cc34 P fc6fa9fee10f418eaafa8c9c58312122: Deleting consensus metadata
I20260929 03:21:22.867009 6747 ts_tablet_manager.cc:1939] T 11f1176973e04097b0efe932c9d1cc34 P 4d156675a70d485eae55a7f5a628a932: Deleting consensus metadata
I20260929 03:21:22.867411 6244 catalog_manager.cc:4841] TS 4d156675a70d485eae55a7f5a628a932 (127.20.178.67:45241): tablet 11f1176973e04097b0efe932c9d1cc34 (table test-table [id=c1a978fcf4294ad9b01fca0e78400867]) successfully deleted
I20260929 03:21:22.867475 6245 catalog_manager.cc:4841] TS fc6fa9fee10f418eaafa8c9c58312122 (127.20.178.66:44491): tablet 11f1176973e04097b0efe932c9d1cc34 (table test-table [id=c1a978fcf4294ad9b01fca0e78400867]) successfully deleted
W20260929 03:21:22.871819 6245 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.20.178.65:40777 (error 108)
I20260929 03:21:22.872069 21193 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuoaHbX/build/release/bin/kudu with pid 6449
I20260929 03:21:22.872956 6716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:22.872254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.65:40777 (local address 127.0.0.1:47272)
0929 03:21:22.872864 (+ 610us) negotiation.cc:107] Waiting for socket to connect
0929 03:21:22.872920 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.178.65:40777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":587}
W20260929 03:21:22.873701 6245 catalog_manager.cc:4584] TS 20f1d5e194234d4a85252e2e64cfaf36 (127.20.178.65:40777): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 11f1176973e04097b0efe932c9d1cc34: Network error: Client connection negotiation failed: client connection to 127.20.178.65:40777: connect: Connection refused (error 111)
I20260929 03:21:22.877782 21193 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuoaHbX/build/release/bin/kudu with pid 6580
I20260929 03:21:22.882691 21193 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuoaHbX/build/release/bin/kudu with pid 6227
2026-09-29T03:21:22Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (696 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651852476208-21193-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:21:22 dist-test-slave-c9qh krb5kdc[6751](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:21:22 dist-test-slave-c9qh krb5kdc[6751](info): set up 2 sockets
Sep 29 03:21:22 dist-test-slave-c9qh krb5kdc[6751](info): commencing operation
krb5kdc: starting...
W20260929 03:21:23.916389 21193 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651852476208-21193-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651852476208-21193-0/krb5kdc/test-user.keytab.
Sep 29 03:21:23 dist-test-slave-c9qh krb5kdc[6751](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652083, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:21:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:21:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.178.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.178.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.178.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.126.keytab.
Entry for principal kudu/127.20.178.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.126.keytab.
I20260929 03:21:24.031752 21193 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuoaHbX/build/release/bin/kudu
/tmp/dist-test-taskuoaHbX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651852476208-21193-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651852476208-21193-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651852476208-21193-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651852476208-21193-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.178.126:42519
--webserver_interface=127.20.178.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.126.keytab
--principal=kudu/127.20.178.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.178.84:38927
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.178.126:42519
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651852476208-21193-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651852476208-21193-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651852476208-21193-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:21:24.111527 6764 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:21:24.111788 6764 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:21:24.111836 6764 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:21:24.113026 6764 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:21:24.113097 6764 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:21:24.113121 6764 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:21:24.113144 6764 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:21:24.113207 6764 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:21:24.114673 6764 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.178.84:38927
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651852476208-21193-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651852476208-21193-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.178.126:42519
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.126.keytab
--principal=kudu/127.20.178.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.178.126:42519
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651852476208-21193-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.178.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6764
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651852476208-21193-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:21:24.114960 6764 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:21:24.115193 6764 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:21:24.119910 6772 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:21:24.119910 6770 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:21:24.120117 6764 server_base.cc:1034] running on GCE node
W20260929 03:21:24.119962 6769 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:21:24.120489 6764 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:21:24.120787 6764 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:21:24.121937 6764 hybrid_clock.cc:648] HybridClock initialized: now 1790652084121924 us; error 28 us; skew 500 ppm
Sep 29 03:21:24 dist-test-slave-c9qh krb5kdc[6751](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652084, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:21:24.124426 6764 init.cc:377] Logged in from keytab as kudu/127.20.178.126@KRBTEST.COM (short username kudu)
I20260929 03:21:24.124924 6764 webserver.cc:460] Webserver started at http://127.20.178.126:33843/ using document root <none> and password file <none>
I20260929 03:21:24.125149 6764 fs_manager.cc:359] Metadata directory not provided
I20260929 03:21:24.125195 6764 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:21:24.125332 6764 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:21:24.126380 6764 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651852476208-21193-0/minicluster-data/master-0/data/instance:
uuid: "549f2664554648d3a298d143a1449633"
format_stamp: "Formatted at 2026-09-29 03:21:24 on dist-test-slave-c9qh"
server_key: "f1737334c5d23f648ebb3fc89582dba3"
server_key_iv: "a868cabee85d1168a895ae479fdc5014"
server_key_version: "encryptionkey@0"
I20260929 03:21:24.126781 6764 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651852476208-21193-0/minicluster-data/master-0/wal/instance:
uuid: "549f2664554648d3a298d143a1449633"
format_stamp: "Formatted at 2026-09-29 03:21:24 on dist-test-slave-c9qh"
server_key: "f1737334c5d23f648ebb3fc89582dba3"
server_key_iv: "a868cabee85d1168a895ae479fdc5014"
server_key_version: "encryptionkey@0"
I20260929 03:21:24.128098 6764 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 03:21:24.128887 6779 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:21:24.129088 6764 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 03:21:24.129163 6764 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651852476208-21193-0/minicluster-data/master-0/data,/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651852476208-21193-0/minicluster-data/master-0/wal
uuid: "549f2664554648d3a298d143a1449633"
format_stamp: "Formatted at 2026-09-29 03:21:24 on dist-test-slave-c9qh"
server_key: "f1737334c5d23f648ebb3fc89582dba3"
server_key_iv: "a868cabee85d1168a895ae479fdc5014"
server_key_version: "encryptionkey@0"
I20260929 03:21:24.129243 6764 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651852476208-21193-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651852476208-21193-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651852476208-21193-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:21:24.140434 6764 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:21:24.142774 6764 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:21:24.142939 6764 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:21:24.146989 6764 rpc_server.cc:307] RPC server started. Bound to: 127.20.178.126:42519
I20260929 03:21:24.147040 6831 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.178.126:42519 every 8 connection(s)
I20260929 03:21:24.147352 6764 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651852476208-21193-0/minicluster-data/master-0/data/info.pb
I20260929 03:21:24.148149 6832 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:21:24.150414 6832 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 549f2664554648d3a298d143a1449633: Bootstrap starting.
I20260929 03:21:24.150970 6832 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 549f2664554648d3a298d143a1449633: Neither blocks nor log segments found. Creating new log.
I20260929 03:21:24.151259 6832 log.cc:824] T 00000000000000000000000000000000 P 549f2664554648d3a298d143a1449633: Log is configured to *not* fsync() on all Append() calls
I20260929 03:21:24.151897 6832 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 549f2664554648d3a298d143a1449633: No bootstrap required, opened a new log
I20260929 03:21:24.153561 6832 raft_consensus.cc:348] T 00000000000000000000000000000000 P 549f2664554648d3a298d143a1449633 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "549f2664554648d3a298d143a1449633" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 42519 } }
I20260929 03:21:24.153714 6832 raft_consensus.cc:374] T 00000000000000000000000000000000 P 549f2664554648d3a298d143a1449633 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:21:24.153764 6832 raft_consensus.cc:729] T 00000000000000000000000000000000 P 549f2664554648d3a298d143a1449633 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 549f2664554648d3a298d143a1449633, State: Initialized, Role: FOLLOWER
I20260929 03:21:24.153927 6832 consensus_queue.cc:260] T 00000000000000000000000000000000 P 549f2664554648d3a298d143a1449633 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "549f2664554648d3a298d143a1449633" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 42519 } }
I20260929 03:21:24.154008 6832 raft_consensus.cc:388] T 00000000000000000000000000000000 P 549f2664554648d3a298d143a1449633 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:21:24.154047 6832 raft_consensus.cc:482] T 00000000000000000000000000000000 P 549f2664554648d3a298d143a1449633 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:21:24.154099 6832 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 549f2664554648d3a298d143a1449633 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:21:24.154608 6832 raft_consensus.cc:504] T 00000000000000000000000000000000 P 549f2664554648d3a298d143a1449633 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "549f2664554648d3a298d143a1449633" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 42519 } }
I20260929 03:21:24.154723 6832 leader_election.cc:302] T 00000000000000000000000000000000 P 549f2664554648d3a298d143a1449633 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 549f2664554648d3a298d143a1449633; no voters:
I20260929 03:21:24.154901 6832 leader_election.cc:288] T 00000000000000000000000000000000 P 549f2664554648d3a298d143a1449633 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:21:24.154925 6835 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 549f2664554648d3a298d143a1449633 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:21:24.155094 6835 raft_consensus.cc:686] T 00000000000000000000000000000000 P 549f2664554648d3a298d143a1449633 [term 1 LEADER]: Becoming Leader. State: Replica: 549f2664554648d3a298d143a1449633, State: Running, Role: LEADER
I20260929 03:21:24.155189 6832 sys_catalog.cc:564] T 00000000000000000000000000000000 P 549f2664554648d3a298d143a1449633 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:21:24.155185 6835 consensus_queue.cc:237] T 00000000000000000000000000000000 P 549f2664554648d3a298d143a1449633 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "549f2664554648d3a298d143a1449633" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 42519 } }
I20260929 03:21:24.155596 6836 sys_catalog.cc:455] T 00000000000000000000000000000000 P 549f2664554648d3a298d143a1449633 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "549f2664554648d3a298d143a1449633" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "549f2664554648d3a298d143a1449633" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 42519 } } }
I20260929 03:21:24.155682 6836 sys_catalog.cc:458] T 00000000000000000000000000000000 P 549f2664554648d3a298d143a1449633 [sys.catalog]: This master's current role is: LEADER
I20260929 03:21:24.155845 6837 sys_catalog.cc:455] T 00000000000000000000000000000000 P 549f2664554648d3a298d143a1449633 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 549f2664554648d3a298d143a1449633. Latest consensus state: current_term: 1 leader_uuid: "549f2664554648d3a298d143a1449633" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "549f2664554648d3a298d143a1449633" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 42519 } } }
I20260929 03:21:24.155941 6837 sys_catalog.cc:458] T 00000000000000000000000000000000 P 549f2664554648d3a298d143a1449633 [sys.catalog]: This master's current role is: LEADER
I20260929 03:21:24.156138 21193 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuoaHbX/build/release/bin/kudu as pid 6764
I20260929 03:21:24.156258 21193 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651852476208-21193-0/minicluster-data/master-0/wal/instance
I20260929 03:21:24.156450 21193 external_mini_cluster.cc:1428] Setting key db59591eeff8154ea49115e2bfa8f189
I20260929 03:21:24.156894 6841 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:21:24.157505 6841 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:21:24.160027 6841 catalog_manager.cc:1347] Generated new cluster ID: 59c4151e2ec944eabf81ba9a17d7b32f
I20260929 03:21:24.160082 6841 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 29 03:21:24 dist-test-slave-c9qh krb5kdc[6751](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652083, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.178.126@KRBTEST.COM
I20260929 03:21:24.167354 6849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:24.156973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48668 (local address 127.20.178.126:42519)
0929 03:21:24.157120 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0929 03:21:24.157130 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 03:21:24.157383 (+ 253us) server_negotiation.cc:407] Connection header received
0929 03:21:24.157523 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:21:24.157528 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:21:24.157600 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:21:24.157762 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:21:24.158532 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:24.159894 (+ 1362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:24.160894 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:24.161316 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:24.161435 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:21:24.163723 (+ 2288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:21:24.163727 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:21:24.163727 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:21:24.163740 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:21:24.165796 (+ 2056us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:21:24.166178 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:21:24.166181 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:21:24.166181 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:21:24.166290 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:21:24.166503 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:21:24.166504 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:21:24.166504 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:21:24.166750 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:21:24.166920 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0929 03:21:24.167017 (+ 97us) server_negotiation.cc:299] Negotiation successful
0929 03:21:24.167157 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97,"thread_start_us":66,"threads_started":1}
I20260929 03:21:24.183598 6841 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:21:24.184338 6841 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:21:24.189904 6841 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 549f2664554648d3a298d143a1449633: Generated new TSK 0
I20260929 03:21:24.190081 6841 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.178.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.178.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.178.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.65.keytab.
Entry for principal kudu/127.20.178.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.65.keytab.
I20260929 03:21:24.215785 21193 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuoaHbX/build/release/bin/kudu
/tmp/dist-test-taskuoaHbX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651852476208-21193-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651852476208-21193-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651852476208-21193-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651852476208-21193-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.178.65:0
--local_ip_for_outbound_sockets=127.20.178.65
--webserver_interface=127.20.178.65
--webserver_port=0
--tserver_master_addrs=127.20.178.126:42519
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.65.keytab
--principal=kudu/127.20.178.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.178.84:38927
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651852476208-21193-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651852476208-21193-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651852476208-21193-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:21:24.295214 6858 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:21:24.295455 6858 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:21:24.295493 6858 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:21:24.296696 6858 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:21:24.296762 6858 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:21:24.296806 6858 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.178.65
I20260929 03:21:24.298261 6858 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.178.84:38927
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651852476208-21193-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651852476208-21193-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.65.keytab
--principal=kudu/127.20.178.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.178.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651852476208-21193-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.178.65
--webserver_port=0
--tserver_master_addrs=127.20.178.126:42519
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6858
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.178.65
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651852476208-21193-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:21:24.298517 6858 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:21:24.298738 6858 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:21:24.303771 6866 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:21:24.303771 6864 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:21:24.303807 6863 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:21:24.303930 6858 server_base.cc:1034] running on GCE node
I20260929 03:21:24.304201 6858 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:21:24.304467 6858 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:21:24.305624 6858 hybrid_clock.cc:648] HybridClock initialized: now 1790652084305602 us; error 55 us; skew 500 ppm
Sep 29 03:21:24 dist-test-slave-c9qh krb5kdc[6751](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652084, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:21:24.308079 6858 init.cc:377] Logged in from keytab as kudu/127.20.178.65@KRBTEST.COM (short username kudu)
I20260929 03:21:24.308490 6858 webserver.cc:460] Webserver started at http://127.20.178.65:39383/ using document root <none> and password file <none>
I20260929 03:21:24.308761 6858 fs_manager.cc:359] Metadata directory not provided
I20260929 03:21:24.308808 6858 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:21:24.308956 6858 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:21:24.310007 6858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651852476208-21193-0/minicluster-data/ts-0/data/instance:
uuid: "aa51f2463acd4b1285537c6d9267d637"
format_stamp: "Formatted at 2026-09-29 03:21:24 on dist-test-slave-c9qh"
server_key: "589989ccbb625736aa97b452796832fd"
server_key_iv: "09f01980efcf0bdd3372804d4739dde7"
server_key_version: "encryptionkey@0"
I20260929 03:21:24.310400 6858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651852476208-21193-0/minicluster-data/ts-0/wal/instance:
uuid: "aa51f2463acd4b1285537c6d9267d637"
format_stamp: "Formatted at 2026-09-29 03:21:24 on dist-test-slave-c9qh"
server_key: "589989ccbb625736aa97b452796832fd"
server_key_iv: "09f01980efcf0bdd3372804d4739dde7"
server_key_version: "encryptionkey@0"
I20260929 03:21:24.311703 6858 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 03:21:24.312441 6873 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:21:24.312609 6858 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 03:21:24.312734 6858 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651852476208-21193-0/minicluster-data/ts-0/data,/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651852476208-21193-0/minicluster-data/ts-0/wal
uuid: "aa51f2463acd4b1285537c6d9267d637"
format_stamp: "Formatted at 2026-09-29 03:21:24 on dist-test-slave-c9qh"
server_key: "589989ccbb625736aa97b452796832fd"
server_key_iv: "09f01980efcf0bdd3372804d4739dde7"
server_key_version: "encryptionkey@0"
I20260929 03:21:24.312819 6858 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651852476208-21193-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651852476208-21193-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651852476208-21193-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:21:24.332571 6858 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:21:24.334738 6858 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:21:24.334911 6858 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:21:24.335156 6858 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:21:24.335487 6858 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:21:24.335543 6858 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:21:24.335579 6858 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:21:24.335606 6858 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:21:24.341337 6858 rpc_server.cc:307] RPC server started. Bound to: 127.20.178.65:42807
I20260929 03:21:24.341441 6986 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.178.65:42807 every 8 connection(s)
I20260929 03:21:24.341745 6858 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651852476208-21193-0/minicluster-data/ts-0/data/info.pb
Sep 29 03:21:24 dist-test-slave-c9qh krb5kdc[6751](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652084, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.65@KRBTEST.COM for kudu/127.20.178.126@KRBTEST.COM
I20260929 03:21:24.350651 6849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:24.342507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.65:48515 (local address 127.20.178.126:42519)
0929 03:21:24.342578 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0929 03:21:24.342580 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:21:24.342789 (+ 209us) server_negotiation.cc:407] Connection header received
0929 03:21:24.343290 (+ 501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:21:24.343291 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:21:24.343302 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:21:24.343361 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:21:24.344301 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:24.344833 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:24.345782 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:24.345979 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:24.346029 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:21:24.348067 (+ 2038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:21:24.348069 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:21:24.348069 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:21:24.348076 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:21:24.349400 (+ 1324us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:21:24.349776 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:21:24.349777 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:21:24.349777 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:21:24.349836 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:21:24.350072 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:21:24.350073 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:21:24.350074 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:21:24.350185 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:21:24.350253 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0929 03:21:24.350484 (+ 231us) server_negotiation.cc:299] Negotiation successful
0929 03:21:24.350559 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260929 03:21:24.350802 21193 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuoaHbX/build/release/bin/kudu as pid 6858
I20260929 03:21:24.350919 21193 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651852476208-21193-0/minicluster-data/ts-0/wal/instance
I20260929 03:21:24.350934 6989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:24.342478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:42519 (local address 127.20.178.65:48515)
0929 03:21:24.342658 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0929 03:21:24.342700 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 03:21:24.343179 (+ 479us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:21:24.343445 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:21:24.343449 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:21:24.344019 (+ 570us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:21:24.344208 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:21:24.344212 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:21:24.344911 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:21:24.344912 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:21:24.345702 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:21:24.345705 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:21:24.346051 (+ 346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:21:24.346052 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:21:24.346148 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:21:24.346577 (+ 429us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:21:24.346585 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:21:24.347972 (+ 1387us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:21:24.349470 (+ 1498us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:21:24.349477 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:21:24.349478 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:21:24.349692 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:21:24.349906 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:21:24.349906 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:21:24.349907 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:21:24.350002 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:21:24.350253 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:21:24.350257 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:21:24.350371 (+ 114us) client_negotiation.cc:769] Sending connection context
0929 03:21:24.350452 (+ 81us) client_negotiation.cc:240] Negotiation successful
0929 03:21:24.350568 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":104,"threads_started":1}
I20260929 03:21:24.351161 21193 external_mini_cluster.cc:1428] Setting key 72b3a3e691487d1c80bd9e78534218d7
I20260929 03:21:24.351405 6987 heartbeater.cc:347] Connected to a master server at 127.20.178.126:42519
I20260929 03:21:24.351514 6987 heartbeater.cc:464] Registering TS with master...
I20260929 03:21:24.351709 6987 heartbeater.cc:511] Master 127.20.178.126:42519 requested a full tablet report, sending...
I20260929 03:21:24.352234 6796 ts_manager.cc:194] Registered new tserver with Master: aa51f2463acd4b1285537c6d9267d637 (127.20.178.65:42807)
I20260929 03:21:24.352937 6796 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.178.65@KRBTEST.COM'} at 127.20.178.65:48515
WARNING: no policy specified for kudu/127.20.178.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.178.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.178.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.66.keytab.
Entry for principal kudu/127.20.178.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.66.keytab.
I20260929 03:21:24.375689 21193 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuoaHbX/build/release/bin/kudu
/tmp/dist-test-taskuoaHbX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651852476208-21193-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651852476208-21193-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651852476208-21193-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651852476208-21193-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.178.66:0
--local_ip_for_outbound_sockets=127.20.178.66
--webserver_interface=127.20.178.66
--webserver_port=0
--tserver_master_addrs=127.20.178.126:42519
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.66.keytab
--principal=kudu/127.20.178.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.178.84:38927
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651852476208-21193-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651852476208-21193-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651852476208-21193-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:21:24.455055 6992 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:21:24.455262 6992 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:21:24.455286 6992 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:21:24.456454 6992 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:21:24.456498 6992 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:21:24.456525 6992 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.178.66
I20260929 03:21:24.457988 6992 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.178.84:38927
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651852476208-21193-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651852476208-21193-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.66.keytab
--principal=kudu/127.20.178.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.178.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651852476208-21193-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.178.66
--webserver_port=0
--tserver_master_addrs=127.20.178.126:42519
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6992
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.178.66
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651852476208-21193-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:21:24.458177 6992 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:21:24.458351 6992 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:21:24.463415 7000 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:21:24.463415 6998 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:21:24.463513 6992 server_base.cc:1034] running on GCE node
W20260929 03:21:24.463444 6997 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:21:24.463912 6992 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:21:24.464179 6992 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:21:24.465322 6992 hybrid_clock.cc:648] HybridClock initialized: now 1790652084465303 us; error 40 us; skew 500 ppm
Sep 29 03:21:24 dist-test-slave-c9qh krb5kdc[6751](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652084, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:21:24.467952 6992 init.cc:377] Logged in from keytab as kudu/127.20.178.66@KRBTEST.COM (short username kudu)
I20260929 03:21:24.468386 6992 webserver.cc:460] Webserver started at http://127.20.178.66:37047/ using document root <none> and password file <none>
I20260929 03:21:24.468609 6992 fs_manager.cc:359] Metadata directory not provided
I20260929 03:21:24.468714 6992 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:21:24.468873 6992 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:21:24.469940 6992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651852476208-21193-0/minicluster-data/ts-1/data/instance:
uuid: "8701413677714ac0ac47fcfd6605169d"
format_stamp: "Formatted at 2026-09-29 03:21:24 on dist-test-slave-c9qh"
server_key: "3534a703b621d411d77c20d5d341599c"
server_key_iv: "ccdcfb827b528e91d7e1c463be871ff2"
server_key_version: "encryptionkey@0"
I20260929 03:21:24.470299 6992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651852476208-21193-0/minicluster-data/ts-1/wal/instance:
uuid: "8701413677714ac0ac47fcfd6605169d"
format_stamp: "Formatted at 2026-09-29 03:21:24 on dist-test-slave-c9qh"
server_key: "3534a703b621d411d77c20d5d341599c"
server_key_iv: "ccdcfb827b528e91d7e1c463be871ff2"
server_key_version: "encryptionkey@0"
I20260929 03:21:24.471647 6992 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 03:21:24.472353 7007 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:21:24.472539 6992 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 03:21:24.472638 6992 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651852476208-21193-0/minicluster-data/ts-1/data,/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651852476208-21193-0/minicluster-data/ts-1/wal
uuid: "8701413677714ac0ac47fcfd6605169d"
format_stamp: "Formatted at 2026-09-29 03:21:24 on dist-test-slave-c9qh"
server_key: "3534a703b621d411d77c20d5d341599c"
server_key_iv: "ccdcfb827b528e91d7e1c463be871ff2"
server_key_version: "encryptionkey@0"
I20260929 03:21:24.472757 6992 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651852476208-21193-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651852476208-21193-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651852476208-21193-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:21:24.496484 6992 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:21:24.498363 6992 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:21:24.498545 6992 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:21:24.498783 6992 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:21:24.499110 6992 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:21:24.499164 6992 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:21:24.499205 6992 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:21:24.499249 6992 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:21:24.505048 6992 rpc_server.cc:307] RPC server started. Bound to: 127.20.178.66:39097
I20260929 03:21:24.505088 7120 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.178.66:39097 every 8 connection(s)
I20260929 03:21:24.505442 6992 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651852476208-21193-0/minicluster-data/ts-1/data/info.pb
I20260929 03:21:24.510639 21193 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuoaHbX/build/release/bin/kudu as pid 6992
I20260929 03:21:24.510768 21193 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651852476208-21193-0/minicluster-data/ts-1/wal/instance
I20260929 03:21:24.510975 21193 external_mini_cluster.cc:1428] Setting key 1f1e8d299c0bfe3bfd560afff96b73b6
Sep 29 03:21:24 dist-test-slave-c9qh krb5kdc[6751](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652084, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.66@KRBTEST.COM for kudu/127.20.178.126@KRBTEST.COM
I20260929 03:21:24.514757 6849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:24.506160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.66:46915 (local address 127.20.178.126:42519)
0929 03:21:24.506256 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0929 03:21:24.506258 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:21:24.506410 (+ 152us) server_negotiation.cc:407] Connection header received
0929 03:21:24.506868 (+ 458us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:21:24.506870 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:21:24.506880 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:21:24.506932 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:21:24.507803 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:24.508289 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:24.509303 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:24.509488 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:24.509542 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:21:24.511685 (+ 2143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:21:24.511687 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:21:24.511687 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:21:24.511695 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:21:24.513395 (+ 1700us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:21:24.513776 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:21:24.513777 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:21:24.513777 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:21:24.513841 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:21:24.514060 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:21:24.514061 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:21:24.514061 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:21:24.514205 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:21:24.514276 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0929 03:21:24.514519 (+ 243us) server_negotiation.cc:299] Negotiation successful
0929 03:21:24.514608 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260929 03:21:24.514856 7123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:24.506144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:42519 (local address 127.20.178.66:46915)
0929 03:21:24.506292 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0929 03:21:24.506326 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0929 03:21:24.506774 (+ 448us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:21:24.506985 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:21:24.506989 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:21:24.507543 (+ 554us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:21:24.507724 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:21:24.507729 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:21:24.508369 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:21:24.508369 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:21:24.509231 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:21:24.509234 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:21:24.509561 (+ 327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:21:24.509562 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:21:24.509659 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:21:24.510072 (+ 413us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:21:24.510080 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:21:24.511589 (+ 1509us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:21:24.513489 (+ 1900us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:21:24.513495 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:21:24.513497 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:21:24.513702 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:21:24.513904 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:21:24.513905 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:21:24.513905 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:21:24.513998 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:21:24.514271 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:21:24.514275 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:21:24.514393 (+ 118us) client_negotiation.cc:769] Sending connection context
0929 03:21:24.514474 (+ 81us) client_negotiation.cc:240] Negotiation successful
0929 03:21:24.514632 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":88,"threads_started":1}
I20260929 03:21:24.515293 7121 heartbeater.cc:347] Connected to a master server at 127.20.178.126:42519
I20260929 03:21:24.515434 7121 heartbeater.cc:464] Registering TS with master...
I20260929 03:21:24.515633 7121 heartbeater.cc:511] Master 127.20.178.126:42519 requested a full tablet report, sending...
I20260929 03:21:24.516010 6796 ts_manager.cc:194] Registered new tserver with Master: 8701413677714ac0ac47fcfd6605169d (127.20.178.66:39097)
I20260929 03:21:24.516604 6796 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.178.66@KRBTEST.COM'} at 127.20.178.66:46915
WARNING: no policy specified for kudu/127.20.178.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.178.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.178.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.67.keytab.
Entry for principal kudu/127.20.178.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.67.keytab.
I20260929 03:21:24.536080 21193 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuoaHbX/build/release/bin/kudu
/tmp/dist-test-taskuoaHbX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651852476208-21193-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651852476208-21193-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651852476208-21193-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651852476208-21193-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.178.67:0
--local_ip_for_outbound_sockets=127.20.178.67
--webserver_interface=127.20.178.67
--webserver_port=0
--tserver_master_addrs=127.20.178.126:42519
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.67.keytab
--principal=kudu/127.20.178.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.178.84:38927
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651852476208-21193-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651852476208-21193-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651852476208-21193-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:21:24.615895 7126 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:21:24.616178 7126 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:21:24.616218 7126 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:21:24.617413 7126 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:21:24.617483 7126 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:21:24.617530 7126 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.178.67
I20260929 03:21:24.619019 7126 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.178.84:38927
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651852476208-21193-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651852476208-21193-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.67.keytab
--principal=kudu/127.20.178.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.178.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651852476208-21193-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.178.67
--webserver_port=0
--tserver_master_addrs=127.20.178.126:42519
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7126
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.178.67
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651852476208-21193-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:21:24.619274 7126 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:21:24.619521 7126 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:21:24.625424 7132 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:21:24.625438 7134 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:21:24.625438 7131 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:21:24.625764 7126 server_base.cc:1034] running on GCE node
I20260929 03:21:24.625977 7126 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:21:24.626220 7126 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:21:24.627388 7126 hybrid_clock.cc:648] HybridClock initialized: now 1790652084627372 us; error 32 us; skew 500 ppm
Sep 29 03:21:24 dist-test-slave-c9qh krb5kdc[6751](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652084, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:21:24.629876 7126 init.cc:377] Logged in from keytab as kudu/127.20.178.67@KRBTEST.COM (short username kudu)
I20260929 03:21:24.630304 7126 webserver.cc:460] Webserver started at http://127.20.178.67:36557/ using document root <none> and password file <none>
I20260929 03:21:24.630533 7126 fs_manager.cc:359] Metadata directory not provided
I20260929 03:21:24.630599 7126 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:21:24.630748 7126 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:21:24.631891 7126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651852476208-21193-0/minicluster-data/ts-2/data/instance:
uuid: "c3cdd0755ad548599032a9c44204b1ef"
format_stamp: "Formatted at 2026-09-29 03:21:24 on dist-test-slave-c9qh"
server_key: "2cfb7de7e5ad1c0f8541d3ca5f42a822"
server_key_iv: "a3578657f59ed732b4552cb3e58163f8"
server_key_version: "encryptionkey@0"
I20260929 03:21:24.632272 7126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651852476208-21193-0/minicluster-data/ts-2/wal/instance:
uuid: "c3cdd0755ad548599032a9c44204b1ef"
format_stamp: "Formatted at 2026-09-29 03:21:24 on dist-test-slave-c9qh"
server_key: "2cfb7de7e5ad1c0f8541d3ca5f42a822"
server_key_iv: "a3578657f59ed732b4552cb3e58163f8"
server_key_version: "encryptionkey@0"
I20260929 03:21:24.633818 7126 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 03:21:24.634552 7141 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:21:24.634717 7126 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260929 03:21:24.634809 7126 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651852476208-21193-0/minicluster-data/ts-2/data,/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651852476208-21193-0/minicluster-data/ts-2/wal
uuid: "c3cdd0755ad548599032a9c44204b1ef"
format_stamp: "Formatted at 2026-09-29 03:21:24 on dist-test-slave-c9qh"
server_key: "2cfb7de7e5ad1c0f8541d3ca5f42a822"
server_key_iv: "a3578657f59ed732b4552cb3e58163f8"
server_key_version: "encryptionkey@0"
I20260929 03:21:24.634866 7126 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651852476208-21193-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651852476208-21193-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651852476208-21193-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:21:24.645723 7126 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:21:24.647620 7126 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:21:24.647725 7126 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:21:24.647902 7126 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:21:24.648211 7126 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:21:24.648267 7126 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:21:24.648306 7126 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:21:24.648330 7126 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:21:24.654306 7126 rpc_server.cc:307] RPC server started. Bound to: 127.20.178.67:33933
I20260929 03:21:24.654357 7254 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.178.67:33933 every 8 connection(s)
I20260929 03:21:24.654639 7126 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651852476208-21193-0/minicluster-data/ts-2/data/info.pb
I20260929 03:21:24.660507 21193 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuoaHbX/build/release/bin/kudu as pid 7126
Sep 29 03:21:24 dist-test-slave-c9qh krb5kdc[6751](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652084, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.67@KRBTEST.COM for kudu/127.20.178.126@KRBTEST.COM
I20260929 03:21:24.660593 21193 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651852476208-21193-0/minicluster-data/ts-2/wal/instance
I20260929 03:21:24.660812 21193 external_mini_cluster.cc:1428] Setting key 06d157cdcf873625af6bf9e075688208
I20260929 03:21:24.663908 6849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:24.655323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.67:53045 (local address 127.20.178.126:42519)
0929 03:21:24.655389 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0929 03:21:24.655391 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:21:24.655597 (+ 206us) server_negotiation.cc:407] Connection header received
0929 03:21:24.656150 (+ 553us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:21:24.656152 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:21:24.656164 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:21:24.656227 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:21:24.657161 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:24.657723 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:24.658707 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:24.658897 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:24.658964 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:21:24.661120 (+ 2156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:21:24.661123 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:21:24.661123 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:21:24.661130 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:21:24.662545 (+ 1415us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:21:24.662944 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:21:24.662945 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:21:24.662945 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:21:24.663008 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:21:24.663275 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:21:24.663276 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:21:24.663276 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:21:24.663394 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:21:24.663465 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0929 03:21:24.663741 (+ 276us) server_negotiation.cc:299] Negotiation successful
0929 03:21:24.663821 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260929 03:21:24.664093 7257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:24.655313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:42519 (local address 127.20.178.67:53045)
0929 03:21:24.655463 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0929 03:21:24.655503 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 03:21:24.656056 (+ 553us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:21:24.656294 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:21:24.656297 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:21:24.656903 (+ 606us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:21:24.657084 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:21:24.657089 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:21:24.657808 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:21:24.657809 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:21:24.658629 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:21:24.658632 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:21:24.658984 (+ 352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:21:24.658985 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:21:24.659076 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:21:24.659619 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:21:24.659627 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:21:24.661054 (+ 1427us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:21:24.662629 (+ 1575us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:21:24.662636 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:21:24.662638 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:21:24.662854 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:21:24.663075 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:21:24.663076 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:21:24.663076 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:21:24.663187 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:21:24.663467 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:21:24.663474 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:21:24.663605 (+ 131us) client_negotiation.cc:769] Sending connection context
0929 03:21:24.663710 (+ 105us) client_negotiation.cc:240] Negotiation successful
0929 03:21:24.663856 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":85,"threads_started":1}
I20260929 03:21:24.664562 7255 heartbeater.cc:347] Connected to a master server at 127.20.178.126:42519
I20260929 03:21:24.664686 7255 heartbeater.cc:464] Registering TS with master...
I20260929 03:21:24.664857 7255 heartbeater.cc:511] Master 127.20.178.126:42519 requested a full tablet report, sending...
I20260929 03:21:24.665186 6796 ts_manager.cc:194] Registered new tserver with Master: c3cdd0755ad548599032a9c44204b1ef (127.20.178.67:33933)
I20260929 03:21:24.665547 6796 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.178.67@KRBTEST.COM'} at 127.20.178.67:53045
I20260929 03:21:24.672683 21193 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 03:21:24.679493 6849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:24.673947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48676 (local address 127.20.178.126:42519)
0929 03:21:24.673988 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0929 03:21:24.673990 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:21:24.674081 (+ 91us) server_negotiation.cc:407] Connection header received
0929 03:21:24.674131 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:21:24.674132 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:21:24.674139 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:21:24.674183 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:21:24.674669 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:24.675105 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:24.675814 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:24.675987 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:24.676038 (+ 51us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:21:24.677339 (+ 1301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:21:24.677340 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:21:24.677340 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:21:24.677345 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:21:24.678451 (+ 1106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:21:24.678786 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:21:24.678787 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:21:24.678787 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:21:24.678834 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:21:24.679022 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:21:24.679022 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:21:24.679023 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:21:24.679163 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:21:24.679267 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0929 03:21:24.679342 (+ 75us) server_negotiation.cc:299] Negotiation successful
0929 03:21:24.679388 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260929 03:21:24.681165 6796 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48676:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 03:21:24.682699 6796 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 03:21:24.688767 7263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:24.684816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.66:39097 (local address 127.0.0.1:52880)
0929 03:21:24.684940 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0929 03:21:24.684979 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 03:21:24.685040 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:21:24.685490 (+ 450us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:21:24.685491 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:21:24.685493 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:21:24.685539 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:21:24.685543 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:21:24.686946 (+ 1403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:21:24.686947 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:21:24.687786 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:21:24.687788 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:21:24.688456 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:21:24.688457 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:21:24.688532 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:21:24.688546 (+ 14us) client_negotiation.cc:769] Sending connection context
0929 03:21:24.688612 (+ 66us) client_negotiation.cc:240] Negotiation successful
0929 03:21:24.688683 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"mutex_wait_us":23,"thread_start_us":65,"threads_started":1}
I20260929 03:21:24.689007 7268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:24.685176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41298 (local address 127.20.178.67:33933)
0929 03:21:24.685348 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0929 03:21:24.685352 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:21:24.685362 (+ 10us) server_negotiation.cc:407] Connection header received
0929 03:21:24.685396 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:21:24.685397 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:21:24.685411 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:21:24.685485 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:21:24.685649 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:24.686429 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:24.687838 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:24.688514 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:24.688580 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:21:24.688652 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0929 03:21:24.688745 (+ 93us) server_negotiation.cc:299] Negotiation successful
0929 03:21:24.688783 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":77,"threads_started":1}
I20260929 03:21:24.689031 7265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:24.684953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52880 (local address 127.20.178.66:39097)
0929 03:21:24.685315 (+ 362us) server_negotiation.cc:206] Beginning negotiation
0929 03:21:24.685319 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:21:24.685327 (+ 8us) server_negotiation.cc:407] Connection header received
0929 03:21:24.685362 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:21:24.685363 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:21:24.685377 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:21:24.685683 (+ 306us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:21:24.685761 (+ 78us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:24.686815 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:24.687875 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:24.688364 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:24.688456 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:21:24.688515 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0929 03:21:24.688896 (+ 381us) server_negotiation.cc:299] Negotiation successful
0929 03:21:24.688949 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":99,"threads_started":1}
I20260929 03:21:24.689251 7266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:24.684886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.65:42807 (local address 127.0.0.1:46962)
0929 03:21:24.685135 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0929 03:21:24.685145 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 03:21:24.685199 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:21:24.685455 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:21:24.685458 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:21:24.685459 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:21:24.685529 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:21:24.685531 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:21:24.686917 (+ 1386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:21:24.686918 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:21:24.687987 (+ 1069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:21:24.687989 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:21:24.688899 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:21:24.688900 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:21:24.688965 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:21:24.688968 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 03:21:24.688999 (+ 31us) client_negotiation.cc:240] Negotiation successful
0929 03:21:24.689041 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":51,"threads_started":1}
I20260929 03:21:24.689311 7267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:24.685084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46962 (local address 127.20.178.65:42807)
0929 03:21:24.685276 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0929 03:21:24.685281 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:21:24.685291 (+ 10us) server_negotiation.cc:407] Connection header received
0929 03:21:24.685342 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:21:24.685344 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:21:24.685360 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:21:24.685444 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:21:24.685608 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:24.686803 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:24.688073 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:24.688806 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:24.689088 (+ 282us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:21:24.689150 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0929 03:21:24.689174 (+ 24us) server_negotiation.cc:299] Negotiation successful
0929 03:21:24.689213 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":84,"threads_started":1}
I20260929 03:21:24.689041 7264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:24.685024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.67:33933 (local address 127.0.0.1:41298)
0929 03:21:24.685262 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0929 03:21:24.685272 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 03:21:24.685321 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:21:24.685549 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:21:24.685549 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:21:24.685551 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:21:24.685587 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:21:24.685589 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:21:24.686538 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:21:24.686538 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:21:24.687746 (+ 1208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:21:24.687749 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:21:24.688594 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:21:24.688595 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:21:24.688672 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:21:24.688675 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 03:21:24.688726 (+ 51us) client_negotiation.cc:240] Negotiation successful
0929 03:21:24.688759 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":36,"threads_started":1}
I20260929 03:21:24.689729 7055 tablet_service.cc:1467] Processing CreateTablet for tablet 8a5abb3d7f37480a84e514e2bfe28375 (DEFAULT_TABLE table=test-table [id=07019f2fc8f04342809a9f351ade57ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:21:24.690069 7055 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a5abb3d7f37480a84e514e2bfe28375. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:21:24.690044 6921 tablet_service.cc:1467] Processing CreateTablet for tablet 8a5abb3d7f37480a84e514e2bfe28375 (DEFAULT_TABLE table=test-table [id=07019f2fc8f04342809a9f351ade57ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:21:24.690387 6921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a5abb3d7f37480a84e514e2bfe28375. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:21:24.692550 7269 tablet_bootstrap.cc:492] T 8a5abb3d7f37480a84e514e2bfe28375 P 8701413677714ac0ac47fcfd6605169d: Bootstrap starting.
I20260929 03:21:24.693001 7270 tablet_bootstrap.cc:492] T 8a5abb3d7f37480a84e514e2bfe28375 P aa51f2463acd4b1285537c6d9267d637: Bootstrap starting.
I20260929 03:21:24.693099 7189 tablet_service.cc:1467] Processing CreateTablet for tablet 8a5abb3d7f37480a84e514e2bfe28375 (DEFAULT_TABLE table=test-table [id=07019f2fc8f04342809a9f351ade57ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:21:24.693455 7189 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a5abb3d7f37480a84e514e2bfe28375. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:21:24.693567 7269 tablet_bootstrap.cc:654] T 8a5abb3d7f37480a84e514e2bfe28375 P 8701413677714ac0ac47fcfd6605169d: Neither blocks nor log segments found. Creating new log.
I20260929 03:21:24.693661 7270 tablet_bootstrap.cc:654] T 8a5abb3d7f37480a84e514e2bfe28375 P aa51f2463acd4b1285537c6d9267d637: Neither blocks nor log segments found. Creating new log.
I20260929 03:21:24.694108 7270 log.cc:824] T 8a5abb3d7f37480a84e514e2bfe28375 P aa51f2463acd4b1285537c6d9267d637: Log is configured to *not* fsync() on all Append() calls
I20260929 03:21:24.694139 7269 log.cc:824] T 8a5abb3d7f37480a84e514e2bfe28375 P 8701413677714ac0ac47fcfd6605169d: Log is configured to *not* fsync() on all Append() calls
I20260929 03:21:24.694804 7270 tablet_bootstrap.cc:492] T 8a5abb3d7f37480a84e514e2bfe28375 P aa51f2463acd4b1285537c6d9267d637: No bootstrap required, opened a new log
I20260929 03:21:24.694885 7270 ts_tablet_manager.cc:1397] T 8a5abb3d7f37480a84e514e2bfe28375 P aa51f2463acd4b1285537c6d9267d637: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 03:21:24.696048 7273 tablet_bootstrap.cc:492] T 8a5abb3d7f37480a84e514e2bfe28375 P c3cdd0755ad548599032a9c44204b1ef: Bootstrap starting.
I20260929 03:21:24.696272 7270 raft_consensus.cc:348] T 8a5abb3d7f37480a84e514e2bfe28375 P aa51f2463acd4b1285537c6d9267d637 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8701413677714ac0ac47fcfd6605169d" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 39097 } } peers { permanent_uuid: "c3cdd0755ad548599032a9c44204b1ef" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 33933 } } peers { permanent_uuid: "aa51f2463acd4b1285537c6d9267d637" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 42807 } }
I20260929 03:21:24.696457 7270 raft_consensus.cc:374] T 8a5abb3d7f37480a84e514e2bfe28375 P aa51f2463acd4b1285537c6d9267d637 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:21:24.696502 7270 raft_consensus.cc:729] T 8a5abb3d7f37480a84e514e2bfe28375 P aa51f2463acd4b1285537c6d9267d637 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa51f2463acd4b1285537c6d9267d637, State: Initialized, Role: FOLLOWER
I20260929 03:21:24.696565 7269 tablet_bootstrap.cc:492] T 8a5abb3d7f37480a84e514e2bfe28375 P 8701413677714ac0ac47fcfd6605169d: No bootstrap required, opened a new log
I20260929 03:21:24.696667 7269 ts_tablet_manager.cc:1397] T 8a5abb3d7f37480a84e514e2bfe28375 P 8701413677714ac0ac47fcfd6605169d: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260929 03:21:24.696614 7270 consensus_queue.cc:260] T 8a5abb3d7f37480a84e514e2bfe28375 P aa51f2463acd4b1285537c6d9267d637 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8701413677714ac0ac47fcfd6605169d" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 39097 } } peers { permanent_uuid: "c3cdd0755ad548599032a9c44204b1ef" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 33933 } } peers { permanent_uuid: "aa51f2463acd4b1285537c6d9267d637" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 42807 } }
I20260929 03:21:24.696909 6987 heartbeater.cc:503] Master 127.20.178.126:42519 was elected leader, sending a full tablet report...
I20260929 03:21:24.697021 7273 tablet_bootstrap.cc:654] T 8a5abb3d7f37480a84e514e2bfe28375 P c3cdd0755ad548599032a9c44204b1ef: Neither blocks nor log segments found. Creating new log.
I20260929 03:21:24.696909 7270 ts_tablet_manager.cc:1428] T 8a5abb3d7f37480a84e514e2bfe28375 P aa51f2463acd4b1285537c6d9267d637: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260929 03:21:24.697574 7273 log.cc:824] T 8a5abb3d7f37480a84e514e2bfe28375 P c3cdd0755ad548599032a9c44204b1ef: Log is configured to *not* fsync() on all Append() calls
I20260929 03:21:24.698278 7273 tablet_bootstrap.cc:492] T 8a5abb3d7f37480a84e514e2bfe28375 P c3cdd0755ad548599032a9c44204b1ef: No bootstrap required, opened a new log
I20260929 03:21:24.698333 7273 ts_tablet_manager.cc:1397] T 8a5abb3d7f37480a84e514e2bfe28375 P c3cdd0755ad548599032a9c44204b1ef: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260929 03:21:24.698616 7269 raft_consensus.cc:348] T 8a5abb3d7f37480a84e514e2bfe28375 P 8701413677714ac0ac47fcfd6605169d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8701413677714ac0ac47fcfd6605169d" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 39097 } } peers { permanent_uuid: "c3cdd0755ad548599032a9c44204b1ef" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 33933 } } peers { permanent_uuid: "aa51f2463acd4b1285537c6d9267d637" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 42807 } }
I20260929 03:21:24.698769 7269 raft_consensus.cc:374] T 8a5abb3d7f37480a84e514e2bfe28375 P 8701413677714ac0ac47fcfd6605169d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:21:24.698837 7269 raft_consensus.cc:729] T 8a5abb3d7f37480a84e514e2bfe28375 P 8701413677714ac0ac47fcfd6605169d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8701413677714ac0ac47fcfd6605169d, State: Initialized, Role: FOLLOWER
I20260929 03:21:24.698974 7269 consensus_queue.cc:260] T 8a5abb3d7f37480a84e514e2bfe28375 P 8701413677714ac0ac47fcfd6605169d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8701413677714ac0ac47fcfd6605169d" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 39097 } } peers { permanent_uuid: "c3cdd0755ad548599032a9c44204b1ef" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 33933 } } peers { permanent_uuid: "aa51f2463acd4b1285537c6d9267d637" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 42807 } }
I20260929 03:21:24.699230 7269 ts_tablet_manager.cc:1428] T 8a5abb3d7f37480a84e514e2bfe28375 P 8701413677714ac0ac47fcfd6605169d: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260929 03:21:24.699265 7121 heartbeater.cc:503] Master 127.20.178.126:42519 was elected leader, sending a full tablet report...
I20260929 03:21:24.700138 7273 raft_consensus.cc:348] T 8a5abb3d7f37480a84e514e2bfe28375 P c3cdd0755ad548599032a9c44204b1ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8701413677714ac0ac47fcfd6605169d" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 39097 } } peers { permanent_uuid: "c3cdd0755ad548599032a9c44204b1ef" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 33933 } } peers { permanent_uuid: "aa51f2463acd4b1285537c6d9267d637" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 42807 } }
I20260929 03:21:24.700307 7273 raft_consensus.cc:374] T 8a5abb3d7f37480a84e514e2bfe28375 P c3cdd0755ad548599032a9c44204b1ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:21:24.700366 7273 raft_consensus.cc:729] T 8a5abb3d7f37480a84e514e2bfe28375 P c3cdd0755ad548599032a9c44204b1ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3cdd0755ad548599032a9c44204b1ef, State: Initialized, Role: FOLLOWER
I20260929 03:21:24.700482 7273 consensus_queue.cc:260] T 8a5abb3d7f37480a84e514e2bfe28375 P c3cdd0755ad548599032a9c44204b1ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8701413677714ac0ac47fcfd6605169d" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 39097 } } peers { permanent_uuid: "c3cdd0755ad548599032a9c44204b1ef" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 33933 } } peers { permanent_uuid: "aa51f2463acd4b1285537c6d9267d637" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 42807 } }
I20260929 03:21:24.700774 7273 ts_tablet_manager.cc:1428] T 8a5abb3d7f37480a84e514e2bfe28375 P c3cdd0755ad548599032a9c44204b1ef: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260929 03:21:24.700809 7255 heartbeater.cc:503] Master 127.20.178.126:42519 was elected leader, sending a full tablet report...
I20260929 03:21:24.743999 7277 raft_consensus.cc:482] T 8a5abb3d7f37480a84e514e2bfe28375 P c3cdd0755ad548599032a9c44204b1ef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:21:24.744127 7277 raft_consensus.cc:504] T 8a5abb3d7f37480a84e514e2bfe28375 P c3cdd0755ad548599032a9c44204b1ef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8701413677714ac0ac47fcfd6605169d" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 39097 } } peers { permanent_uuid: "c3cdd0755ad548599032a9c44204b1ef" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 33933 } } peers { permanent_uuid: "aa51f2463acd4b1285537c6d9267d637" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 42807 } }
I20260929 03:21:24.744472 7277 leader_election.cc:288] T 8a5abb3d7f37480a84e514e2bfe28375 P c3cdd0755ad548599032a9c44204b1ef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8701413677714ac0ac47fcfd6605169d (127.20.178.66:39097), aa51f2463acd4b1285537c6d9267d637 (127.20.178.65:42807)
I20260929 03:21:24.747378 7257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:24.744615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.65:42807 (local address 127.20.178.67:36019)
0929 03:21:24.744693 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0929 03:21:24.744706 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 03:21:24.744768 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:21:24.744932 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:21:24.744933 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:21:24.744934 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:21:24.745013 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:21:24.745015 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:21:24.745705 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:21:24.745706 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:21:24.746514 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:21:24.746516 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:21:24.747111 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:21:24.747112 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:21:24.747202 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:21:24.747205 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 03:21:24.747260 (+ 55us) client_negotiation.cc:240] Negotiation successful
0929 03:21:24.747303 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260929 03:21:24.747401 7267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:24.744715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.67:36019 (local address 127.20.178.65:42807)
0929 03:21:24.744763 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0929 03:21:24.744765 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:21:24.744774 (+ 9us) server_negotiation.cc:407] Connection header received
0929 03:21:24.744827 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:21:24.744828 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:21:24.744855 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:21:24.744915 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:21:24.745092 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:24.745615 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:24.746608 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:24.747034 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:24.747091 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:21:24.747098 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0929 03:21:24.747277 (+ 179us) server_negotiation.cc:299] Negotiation successful
0929 03:21:24.747315 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260929 03:21:24.747517 7265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:24.744716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.67:43569 (local address 127.20.178.66:39097)
0929 03:21:24.744759 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0929 03:21:24.744761 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:21:24.744941 (+ 180us) server_negotiation.cc:407] Connection header received
0929 03:21:24.744971 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:21:24.744972 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:21:24.744985 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:21:24.745036 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:21:24.745278 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:24.745801 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:24.746738 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:24.747137 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:24.747207 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:21:24.747215 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0929 03:21:24.747406 (+ 191us) server_negotiation.cc:299] Negotiation successful
0929 03:21:24.747446 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260929 03:21:24.747596 7278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:24.744625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.66:39097 (local address 127.20.178.67:43569)
0929 03:21:24.744833 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0929 03:21:24.744843 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 03:21:24.744905 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:21:24.745076 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:21:24.745077 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:21:24.745079 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:21:24.745153 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:21:24.745155 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:21:24.745890 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:21:24.745890 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:21:24.746644 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:21:24.746646 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:21:24.747223 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:21:24.747223 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:21:24.747311 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:21:24.747315 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 03:21:24.747382 (+ 67us) client_negotiation.cc:240] Negotiation successful
0929 03:21:24.747418 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"mutex_wait_us":1,"thread_start_us":106,"threads_started":1}
I20260929 03:21:24.747972 7075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a5abb3d7f37480a84e514e2bfe28375" candidate_uuid: "c3cdd0755ad548599032a9c44204b1ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8701413677714ac0ac47fcfd6605169d" is_pre_election: true
I20260929 03:21:24.748065 6941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a5abb3d7f37480a84e514e2bfe28375" candidate_uuid: "c3cdd0755ad548599032a9c44204b1ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa51f2463acd4b1285537c6d9267d637" is_pre_election: true
I20260929 03:21:24.748142 7075 raft_consensus.cc:2454] T 8a5abb3d7f37480a84e514e2bfe28375 P 8701413677714ac0ac47fcfd6605169d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3cdd0755ad548599032a9c44204b1ef in term 0.
I20260929 03:21:24.748183 6941 raft_consensus.cc:2454] T 8a5abb3d7f37480a84e514e2bfe28375 P aa51f2463acd4b1285537c6d9267d637 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3cdd0755ad548599032a9c44204b1ef in term 0.
I20260929 03:21:24.748411 7142 leader_election.cc:302] T 8a5abb3d7f37480a84e514e2bfe28375 P c3cdd0755ad548599032a9c44204b1ef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8701413677714ac0ac47fcfd6605169d, c3cdd0755ad548599032a9c44204b1ef; no voters:
I20260929 03:21:24.748551 7277 raft_consensus.cc:2781] T 8a5abb3d7f37480a84e514e2bfe28375 P c3cdd0755ad548599032a9c44204b1ef [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 03:21:24.748615 7277 raft_consensus.cc:482] T 8a5abb3d7f37480a84e514e2bfe28375 P c3cdd0755ad548599032a9c44204b1ef [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 03:21:24.748689 7277 raft_consensus.cc:3037] T 8a5abb3d7f37480a84e514e2bfe28375 P c3cdd0755ad548599032a9c44204b1ef [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:21:24.749362 7277 raft_consensus.cc:504] T 8a5abb3d7f37480a84e514e2bfe28375 P c3cdd0755ad548599032a9c44204b1ef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8701413677714ac0ac47fcfd6605169d" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 39097 } } peers { permanent_uuid: "c3cdd0755ad548599032a9c44204b1ef" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 33933 } } peers { permanent_uuid: "aa51f2463acd4b1285537c6d9267d637" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 42807 } }
I20260929 03:21:24.749507 7277 leader_election.cc:288] T 8a5abb3d7f37480a84e514e2bfe28375 P c3cdd0755ad548599032a9c44204b1ef [CANDIDATE]: Term 1 election: Requested vote from peers 8701413677714ac0ac47fcfd6605169d (127.20.178.66:39097), aa51f2463acd4b1285537c6d9267d637 (127.20.178.65:42807)
I20260929 03:21:24.749663 7075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a5abb3d7f37480a84e514e2bfe28375" candidate_uuid: "c3cdd0755ad548599032a9c44204b1ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8701413677714ac0ac47fcfd6605169d"
I20260929 03:21:24.749663 6941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a5abb3d7f37480a84e514e2bfe28375" candidate_uuid: "c3cdd0755ad548599032a9c44204b1ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa51f2463acd4b1285537c6d9267d637"
I20260929 03:21:24.749750 6941 raft_consensus.cc:3037] T 8a5abb3d7f37480a84e514e2bfe28375 P aa51f2463acd4b1285537c6d9267d637 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:21:24.749765 7075 raft_consensus.cc:3037] T 8a5abb3d7f37480a84e514e2bfe28375 P 8701413677714ac0ac47fcfd6605169d [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:21:24.750674 6941 raft_consensus.cc:2454] T 8a5abb3d7f37480a84e514e2bfe28375 P aa51f2463acd4b1285537c6d9267d637 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3cdd0755ad548599032a9c44204b1ef in term 1.
I20260929 03:21:24.750674 7075 raft_consensus.cc:2454] T 8a5abb3d7f37480a84e514e2bfe28375 P 8701413677714ac0ac47fcfd6605169d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3cdd0755ad548599032a9c44204b1ef in term 1.
I20260929 03:21:24.750857 7144 leader_election.cc:302] T 8a5abb3d7f37480a84e514e2bfe28375 P c3cdd0755ad548599032a9c44204b1ef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aa51f2463acd4b1285537c6d9267d637, c3cdd0755ad548599032a9c44204b1ef; no voters:
I20260929 03:21:24.750947 7277 raft_consensus.cc:2781] T 8a5abb3d7f37480a84e514e2bfe28375 P c3cdd0755ad548599032a9c44204b1ef [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:21:24.751092 7277 raft_consensus.cc:686] T 8a5abb3d7f37480a84e514e2bfe28375 P c3cdd0755ad548599032a9c44204b1ef [term 1 LEADER]: Becoming Leader. State: Replica: c3cdd0755ad548599032a9c44204b1ef, State: Running, Role: LEADER
I20260929 03:21:24.751204 7277 consensus_queue.cc:237] T 8a5abb3d7f37480a84e514e2bfe28375 P c3cdd0755ad548599032a9c44204b1ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8701413677714ac0ac47fcfd6605169d" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 39097 } } peers { permanent_uuid: "c3cdd0755ad548599032a9c44204b1ef" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 33933 } } peers { permanent_uuid: "aa51f2463acd4b1285537c6d9267d637" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 42807 } }
I20260929 03:21:24.752007 6796 catalog_manager.cc:5495] T 8a5abb3d7f37480a84e514e2bfe28375 P c3cdd0755ad548599032a9c44204b1ef reported cstate change: term changed from 0 to 1, leader changed from <none> to c3cdd0755ad548599032a9c44204b1ef (127.20.178.67). New cstate: current_term: 1 leader_uuid: "c3cdd0755ad548599032a9c44204b1ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8701413677714ac0ac47fcfd6605169d" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 39097 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c3cdd0755ad548599032a9c44204b1ef" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 33933 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aa51f2463acd4b1285537c6d9267d637" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 42807 } health_report { overall_health: UNKNOWN } } }
W20260929 03:21:24.755949 7122 tablet.cc:2369] T 8a5abb3d7f37480a84e514e2bfe28375 P 8701413677714ac0ac47fcfd6605169d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 03:21:24.761628 7268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:24.759369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41300 (local address 127.20.178.67:33933)
0929 03:21:24.759413 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0929 03:21:24.759414 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 03:21:24.759422 (+ 8us) server_negotiation.cc:407] Connection header received
0929 03:21:24.759495 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:21:24.759496 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:21:24.759510 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:21:24.759566 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:21:24.759680 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:24.760113 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:24.760888 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:24.761056 (+ 168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:24.761107 (+ 51us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:21:24.761231 (+ 124us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:21:24.761308 (+ 77us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:21:24.761381 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 03:21:24.761493 (+ 112us) server_negotiation.cc:299] Negotiation successful
0929 03:21:24.761501 (+ 8us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260929 03:21:24.763381 7075 raft_consensus.cc:1261] T 8a5abb3d7f37480a84e514e2bfe28375 P 8701413677714ac0ac47fcfd6605169d [term 1 FOLLOWER]: Refusing update from remote peer c3cdd0755ad548599032a9c44204b1ef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:21:24.763381 6941 raft_consensus.cc:1261] T 8a5abb3d7f37480a84e514e2bfe28375 P aa51f2463acd4b1285537c6d9267d637 [term 1 FOLLOWER]: Refusing update from remote peer c3cdd0755ad548599032a9c44204b1ef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:21:24.763729 7277 consensus_queue.cc:1035] T 8a5abb3d7f37480a84e514e2bfe28375 P c3cdd0755ad548599032a9c44204b1ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "8701413677714ac0ac47fcfd6605169d" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 39097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:21:24.763867 7279 consensus_queue.cc:1035] T 8a5abb3d7f37480a84e514e2bfe28375 P c3cdd0755ad548599032a9c44204b1ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa51f2463acd4b1285537c6d9267d637" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 42807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:21:24.766969 7284 mvcc.cc:205] Tried to move back new op lower bound from 7334510939188162560 to 7334510939141574656. Current Snapshot: MvccSnapshot[applied={T|T < 7334510939188162560}]
I20260929 03:21:24.768517 6795 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48676:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:21:24.768625 6795 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48676:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:21:24.769393 6795 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 549f2664554648d3a298d143a1449633: Sending DeleteTablet for 3 replicas of tablet 8a5abb3d7f37480a84e514e2bfe28375
I20260929 03:21:24.769718 7055 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a5abb3d7f37480a84e514e2bfe28375 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:21:24 UTC) from {username='kudu', principal='kudu/127.20.178.126@KRBTEST.COM'} at 127.0.0.1:52880
I20260929 03:21:24.769749 7189 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a5abb3d7f37480a84e514e2bfe28375 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:21:24 UTC) from {username='kudu', principal='kudu/127.20.178.126@KRBTEST.COM'} at 127.0.0.1:41298
I20260929 03:21:24.769850 6921 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a5abb3d7f37480a84e514e2bfe28375 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:21:24 UTC) from {username='kudu', principal='kudu/127.20.178.126@KRBTEST.COM'} at 127.0.0.1:46962
I20260929 03:21:24.769935 7291 tablet_replica.cc:330] T 8a5abb3d7f37480a84e514e2bfe28375 P 8701413677714ac0ac47fcfd6605169d: stopping tablet replica
I20260929 03:21:24.769980 7292 tablet_replica.cc:330] T 8a5abb3d7f37480a84e514e2bfe28375 P c3cdd0755ad548599032a9c44204b1ef: stopping tablet replica
I20260929 03:21:24.770023 7291 raft_consensus.cc:2229] T 8a5abb3d7f37480a84e514e2bfe28375 P 8701413677714ac0ac47fcfd6605169d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:21:24.770066 7292 raft_consensus.cc:2229] T 8a5abb3d7f37480a84e514e2bfe28375 P c3cdd0755ad548599032a9c44204b1ef [term 1 LEADER]: Raft consensus shutting down.
I20260929 03:21:24.770117 7291 raft_consensus.cc:2258] T 8a5abb3d7f37480a84e514e2bfe28375 P 8701413677714ac0ac47fcfd6605169d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:21:24.770145 7292 raft_consensus.cc:2258] T 8a5abb3d7f37480a84e514e2bfe28375 P c3cdd0755ad548599032a9c44204b1ef [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:21:24.770462 21193 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuoaHbX/build/release/bin/kudu with pid 6858
I20260929 03:21:24.771028 7292 ts_tablet_manager.cc:1905] T 8a5abb3d7f37480a84e514e2bfe28375 P c3cdd0755ad548599032a9c44204b1ef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:21:24.772811 7292 ts_tablet_manager.cc:1918] T 8a5abb3d7f37480a84e514e2bfe28375 P c3cdd0755ad548599032a9c44204b1ef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:21:24.772897 7292 log.cc:1196] T 8a5abb3d7f37480a84e514e2bfe28375 P c3cdd0755ad548599032a9c44204b1ef: Deleting WAL directory at /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651852476208-21193-0/minicluster-data/ts-2/wal/wals/8a5abb3d7f37480a84e514e2bfe28375
I20260929 03:21:24.773085 7291 ts_tablet_manager.cc:1905] T 8a5abb3d7f37480a84e514e2bfe28375 P 8701413677714ac0ac47fcfd6605169d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:21:24.773183 7292 ts_tablet_manager.cc:1939] T 8a5abb3d7f37480a84e514e2bfe28375 P c3cdd0755ad548599032a9c44204b1ef: Deleting consensus metadata
I20260929 03:21:24.773640 6780 catalog_manager.cc:4841] TS c3cdd0755ad548599032a9c44204b1ef (127.20.178.67:33933): tablet 8a5abb3d7f37480a84e514e2bfe28375 (table test-table [id=07019f2fc8f04342809a9f351ade57ec]) successfully deleted
I20260929 03:21:24.774914 7291 ts_tablet_manager.cc:1918] T 8a5abb3d7f37480a84e514e2bfe28375 P 8701413677714ac0ac47fcfd6605169d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:21:24.774987 7291 log.cc:1196] T 8a5abb3d7f37480a84e514e2bfe28375 P 8701413677714ac0ac47fcfd6605169d: Deleting WAL directory at /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651852476208-21193-0/minicluster-data/ts-1/wal/wals/8a5abb3d7f37480a84e514e2bfe28375
I20260929 03:21:24.775273 7291 ts_tablet_manager.cc:1939] T 8a5abb3d7f37480a84e514e2bfe28375 P 8701413677714ac0ac47fcfd6605169d: Deleting consensus metadata
I20260929 03:21:24.775815 6780 catalog_manager.cc:4841] TS 8701413677714ac0ac47fcfd6605169d (127.20.178.66:39097): tablet 8a5abb3d7f37480a84e514e2bfe28375 (table test-table [id=07019f2fc8f04342809a9f351ade57ec]) successfully deleted
W20260929 03:21:24.779635 6782 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.20.178.65:42807 (error 108)
I20260929 03:21:24.779841 21193 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuoaHbX/build/release/bin/kudu with pid 6992
I20260929 03:21:24.780160 7264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:24.780005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.65:42807 (local address 127.0.0.1:46976)
0929 03:21:24.780090 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0929 03:21:24.780124 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.178.65:42807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260929 03:21:24.780540 6782 catalog_manager.cc:4584] TS aa51f2463acd4b1285537c6d9267d637 (127.20.178.65:42807): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8a5abb3d7f37480a84e514e2bfe28375: Network error: Client connection negotiation failed: client connection to 127.20.178.65:42807: connect: Connection refused (error 111)
I20260929 03:21:24.784832 21193 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuoaHbX/build/release/bin/kudu with pid 7126
I20260929 03:21:24.790241 21193 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuoaHbX/build/release/bin/kudu with pid 6764
2026-09-29T03:21:24Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (1908 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-29T03:21:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:21:24Z Disabled control of system clock
I20260929 03:21:24.816987 21193 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuoaHbX/build/release/bin/kudu
/tmp/dist-test-taskuoaHbX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790651852476208-21193-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790651852476208-21193-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790651852476208-21193-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790651852476208-21193-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.178.126:40469
--webserver_interface=127.20.178.126
--webserver_port=0
--builtin_ntp_servers=127.20.178.84:41283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.178.126:40469
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 03:21:24.896874 7298 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:21:24.897085 7298 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:21:24.897105 7298 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:21:24.898168 7298 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:21:24.898201 7298 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:21:24.898213 7298 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:21:24.898233 7298 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:21:24.898245 7298 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:21:24.899484 7298 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.178.84:41283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790651852476208-21193-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790651852476208-21193-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.178.126:40469
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.178.126:40469
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790651852476208-21193-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.178.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7298
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790651852476208-21193-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:21:24.899669 7298 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:21:24.899905 7298 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:21:24.904582 7303 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:21:24.904649 7304 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:21:24.904593 7306 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:21:24.904798 7298 server_base.cc:1034] running on GCE node
I20260929 03:21:24.905061 7298 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:21:24.905408 7298 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:21:24.906556 7298 hybrid_clock.cc:648] HybridClock initialized: now 1790652084906548 us; error 29 us; skew 500 ppm
I20260929 03:21:24.907718 7298 webserver.cc:460] Webserver started at http://127.20.178.126:39843/ using document root <none> and password file <none>
I20260929 03:21:24.907964 7298 fs_manager.cc:359] Metadata directory not provided
I20260929 03:21:24.908010 7298 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:21:24.908156 7298 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:21:24.909287 7298 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790651852476208-21193-0/minicluster-data/master-0/data/instance:
uuid: "ee6f31141ac54883b1348b436e10df2c"
format_stamp: "Formatted at 2026-09-29 03:21:24 on dist-test-slave-c9qh"
server_key: "9f6562f375a82eb03a4e6a50bc342d13"
server_key_iv: "fd9579d5b9f2b51388472387b80ba8db"
server_key_version: "encryptionkey@0"
I20260929 03:21:24.909641 7298 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790651852476208-21193-0/minicluster-data/master-0/wal/instance:
uuid: "ee6f31141ac54883b1348b436e10df2c"
format_stamp: "Formatted at 2026-09-29 03:21:24 on dist-test-slave-c9qh"
server_key: "9f6562f375a82eb03a4e6a50bc342d13"
server_key_iv: "fd9579d5b9f2b51388472387b80ba8db"
server_key_version: "encryptionkey@0"
I20260929 03:21:24.911024 7298 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 03:21:24.911859 7312 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:21:24.912036 7298 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 03:21:24.912153 7298 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790651852476208-21193-0/minicluster-data/master-0/data,/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790651852476208-21193-0/minicluster-data/master-0/wal
uuid: "ee6f31141ac54883b1348b436e10df2c"
format_stamp: "Formatted at 2026-09-29 03:21:24 on dist-test-slave-c9qh"
server_key: "9f6562f375a82eb03a4e6a50bc342d13"
server_key_iv: "fd9579d5b9f2b51388472387b80ba8db"
server_key_version: "encryptionkey@0"
I20260929 03:21:24.912271 7298 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790651852476208-21193-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790651852476208-21193-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790651852476208-21193-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:21:24.924569 7298 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:21:24.924933 7298 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:21:24.925081 7298 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:21:24.928895 7298 rpc_server.cc:307] RPC server started. Bound to: 127.20.178.126:40469
I20260929 03:21:24.928956 7364 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.178.126:40469 every 8 connection(s)
I20260929 03:21:24.929266 7298 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790651852476208-21193-0/minicluster-data/master-0/data/info.pb
I20260929 03:21:24.930164 7365 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:21:24.930895 21193 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuoaHbX/build/release/bin/kudu as pid 7298
I20260929 03:21:24.931011 21193 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790651852476208-21193-0/minicluster-data/master-0/wal/instance
I20260929 03:21:24.931216 21193 external_mini_cluster.cc:1428] Setting key b54f48d95f82049a1064407a961e0739
I20260929 03:21:24.933259 7365 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee6f31141ac54883b1348b436e10df2c: Bootstrap starting.
I20260929 03:21:24.933902 7365 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ee6f31141ac54883b1348b436e10df2c: Neither blocks nor log segments found. Creating new log.
I20260929 03:21:24.934201 7365 log.cc:824] T 00000000000000000000000000000000 P ee6f31141ac54883b1348b436e10df2c: Log is configured to *not* fsync() on all Append() calls
I20260929 03:21:24.934966 7365 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee6f31141ac54883b1348b436e10df2c: No bootstrap required, opened a new log
I20260929 03:21:24.936786 7368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:24.931763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46870 (local address 127.20.178.126:40469)
0929 03:21:24.931947 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0929 03:21:24.931954 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:21:24.931978 (+ 24us) server_negotiation.cc:407] Connection header received
0929 03:21:24.932424 (+ 446us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:21:24.932430 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:21:24.932488 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:21:24.932597 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:21:24.932870 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:24.933735 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:24.934829 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:24.935291 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:24.935385 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:21:24.936057 (+ 672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:21:24.936061 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:21:24.936061 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:21:24.936243 (+ 182us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:21:24.936274 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:21:24.936284 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:21:24.936333 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0929 03:21:24.936437 (+ 104us) server_negotiation.cc:299] Negotiation successful
0929 03:21:24.936518 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":77,"threads_started":1}
I20260929 03:21:24.937348 7365 raft_consensus.cc:348] T 00000000000000000000000000000000 P ee6f31141ac54883b1348b436e10df2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee6f31141ac54883b1348b436e10df2c" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 40469 } }
I20260929 03:21:24.937474 7365 raft_consensus.cc:374] T 00000000000000000000000000000000 P ee6f31141ac54883b1348b436e10df2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:21:24.937518 7365 raft_consensus.cc:729] T 00000000000000000000000000000000 P ee6f31141ac54883b1348b436e10df2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee6f31141ac54883b1348b436e10df2c, State: Initialized, Role: FOLLOWER
I20260929 03:21:24.937642 7365 consensus_queue.cc:260] T 00000000000000000000000000000000 P ee6f31141ac54883b1348b436e10df2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee6f31141ac54883b1348b436e10df2c" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 40469 } }
I20260929 03:21:24.937734 7365 raft_consensus.cc:388] T 00000000000000000000000000000000 P ee6f31141ac54883b1348b436e10df2c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:21:24.937762 7365 raft_consensus.cc:482] T 00000000000000000000000000000000 P ee6f31141ac54883b1348b436e10df2c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:21:24.937794 7365 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ee6f31141ac54883b1348b436e10df2c [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:21:24.938251 7365 raft_consensus.cc:504] T 00000000000000000000000000000000 P ee6f31141ac54883b1348b436e10df2c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee6f31141ac54883b1348b436e10df2c" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 40469 } }
I20260929 03:21:24.938342 7365 leader_election.cc:302] T 00000000000000000000000000000000 P ee6f31141ac54883b1348b436e10df2c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ee6f31141ac54883b1348b436e10df2c; no voters:
I20260929 03:21:24.938489 7365 leader_election.cc:288] T 00000000000000000000000000000000 P ee6f31141ac54883b1348b436e10df2c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:21:24.938522 7370 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ee6f31141ac54883b1348b436e10df2c [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:21:24.938627 7370 raft_consensus.cc:686] T 00000000000000000000000000000000 P ee6f31141ac54883b1348b436e10df2c [term 1 LEADER]: Becoming Leader. State: Replica: ee6f31141ac54883b1348b436e10df2c, State: Running, Role: LEADER
I20260929 03:21:24.938710 7370 consensus_queue.cc:237] T 00000000000000000000000000000000 P ee6f31141ac54883b1348b436e10df2c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee6f31141ac54883b1348b436e10df2c" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 40469 } }
I20260929 03:21:24.938751 7365 sys_catalog.cc:564] T 00000000000000000000000000000000 P ee6f31141ac54883b1348b436e10df2c [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:21:24.939034 7371 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee6f31141ac54883b1348b436e10df2c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ee6f31141ac54883b1348b436e10df2c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee6f31141ac54883b1348b436e10df2c" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 40469 } } }
I20260929 03:21:24.939034 7372 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee6f31141ac54883b1348b436e10df2c [sys.catalog]: SysCatalogTable state changed. Reason: New leader ee6f31141ac54883b1348b436e10df2c. Latest consensus state: current_term: 1 leader_uuid: "ee6f31141ac54883b1348b436e10df2c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee6f31141ac54883b1348b436e10df2c" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 40469 } } }
I20260929 03:21:24.939153 7371 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee6f31141ac54883b1348b436e10df2c [sys.catalog]: This master's current role is: LEADER
I20260929 03:21:24.939162 7372 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee6f31141ac54883b1348b436e10df2c [sys.catalog]: This master's current role is: LEADER
I20260929 03:21:24.940008 7377 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:21:24.940609 7377 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:21:24.942040 7377 catalog_manager.cc:1347] Generated new cluster ID: 627fb5af084249d593dd0bb6c60f68a3
I20260929 03:21:24.942090 7377 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:21:24.946472 7377 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:21:24.947158 7377 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:21:24.956966 7377 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ee6f31141ac54883b1348b436e10df2c: Generated new TSK 0
I20260929 03:21:24.957127 7377 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 03:21:24.967188 21193 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuoaHbX/build/release/bin/kudu
/tmp/dist-test-taskuoaHbX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790651852476208-21193-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790651852476208-21193-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790651852476208-21193-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790651852476208-21193-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.178.65:0
--local_ip_for_outbound_sockets=127.20.178.65
--webserver_interface=127.20.178.65
--webserver_port=0
--tserver_master_addrs=127.20.178.126:40469
--builtin_ntp_servers=127.20.178.84:41283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 03:21:25.047264 7389 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:21:25.047448 7389 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:21:25.047469 7389 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:21:25.048591 7389 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:21:25.048653 7389 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:21:25.048717 7389 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.178.65
I20260929 03:21:25.050077 7389 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.178.84:41283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790651852476208-21193-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790651852476208-21193-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.178.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790651852476208-21193-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.178.65
--webserver_port=0
--tserver_master_addrs=127.20.178.126:40469
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7389
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.178.65
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790651852476208-21193-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:21:25.050318 7389 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:21:25.050606 7389 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:21:25.055657 7397 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:21:25.055634 7395 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:21:25.055629 7394 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:21:25.055962 7389 server_base.cc:1034] running on GCE node
I20260929 03:21:25.056160 7389 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:21:25.056411 7389 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:21:25.057570 7389 hybrid_clock.cc:648] HybridClock initialized: now 1790652085057539 us; error 58 us; skew 500 ppm
I20260929 03:21:25.058894 7389 webserver.cc:460] Webserver started at http://127.20.178.65:44037/ using document root <none> and password file <none>
I20260929 03:21:25.059118 7389 fs_manager.cc:359] Metadata directory not provided
I20260929 03:21:25.059183 7389 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:21:25.059355 7389 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:21:25.060384 7389 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790651852476208-21193-0/minicluster-data/ts-0/data/instance:
uuid: "199fe8c8fad84ac298b6c12756afd961"
format_stamp: "Formatted at 2026-09-29 03:21:25 on dist-test-slave-c9qh"
server_key: "081ece9144926daeb5ef0537210ab08a"
server_key_iv: "607fc446a649f2d39d006fcb48db8328"
server_key_version: "encryptionkey@0"
I20260929 03:21:25.060775 7389 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790651852476208-21193-0/minicluster-data/ts-0/wal/instance:
uuid: "199fe8c8fad84ac298b6c12756afd961"
format_stamp: "Formatted at 2026-09-29 03:21:25 on dist-test-slave-c9qh"
server_key: "081ece9144926daeb5ef0537210ab08a"
server_key_iv: "607fc446a649f2d39d006fcb48db8328"
server_key_version: "encryptionkey@0"
I20260929 03:21:25.062136 7389 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260929 03:21:25.062887 7403 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:21:25.063118 7389 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260929 03:21:25.063212 7389 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790651852476208-21193-0/minicluster-data/ts-0/data,/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790651852476208-21193-0/minicluster-data/ts-0/wal
uuid: "199fe8c8fad84ac298b6c12756afd961"
format_stamp: "Formatted at 2026-09-29 03:21:25 on dist-test-slave-c9qh"
server_key: "081ece9144926daeb5ef0537210ab08a"
server_key_iv: "607fc446a649f2d39d006fcb48db8328"
server_key_version: "encryptionkey@0"
I20260929 03:21:25.063282 7389 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790651852476208-21193-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790651852476208-21193-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790651852476208-21193-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:21:25.075933 7389 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:21:25.076176 7389 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:21:25.076275 7389 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:21:25.076449 7389 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:21:25.076853 7389 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:21:25.076891 7389 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:21:25.076913 7389 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:21:25.076926 7389 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:21:25.082487 7389 rpc_server.cc:307] RPC server started. Bound to: 127.20.178.65:39185
I20260929 03:21:25.082561 7516 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.178.65:39185 every 8 connection(s)
I20260929 03:21:25.082907 7389 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790651852476208-21193-0/minicluster-data/ts-0/data/info.pb
I20260929 03:21:25.088166 7368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:25.083662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.65:46127 (local address 127.20.178.126:40469)
0929 03:21:25.083706 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0929 03:21:25.083708 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:21:25.084001 (+ 293us) server_negotiation.cc:407] Connection header received
0929 03:21:25.084533 (+ 532us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:21:25.084534 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:21:25.084549 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:21:25.084599 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:21:25.085298 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:25.085803 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:25.086796 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:25.086995 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:25.087070 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:21:25.087866 (+ 796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:21:25.087867 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:21:25.087868 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:21:25.087872 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:21:25.087888 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:21:25.087892 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:21:25.087962 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0929 03:21:25.088057 (+ 95us) server_negotiation.cc:299] Negotiation successful
0929 03:21:25.088086 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260929 03:21:25.088421 7519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:25.083632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:40469 (local address 127.20.178.65:46127)
0929 03:21:25.083873 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0929 03:21:25.083907 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0929 03:21:25.084424 (+ 517us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:21:25.084705 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:21:25.084709 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:21:25.084717 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:21:25.085195 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:21:25.085199 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:21:25.085906 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:21:25.085907 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:21:25.086687 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:21:25.086689 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:21:25.087094 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:21:25.087095 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:21:25.087223 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:21:25.087723 (+ 500us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 03:21:25.087724 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:21:25.087750 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 03:21:25.087772 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:21:25.087963 (+ 191us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:21:25.087970 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:21:25.087973 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 03:21:25.088060 (+ 87us) client_negotiation.cc:240] Negotiation successful
0929 03:21:25.088161 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":97,"threads_started":1}
I20260929 03:21:25.089028 7517 heartbeater.cc:347] Connected to a master server at 127.20.178.126:40469
I20260929 03:21:25.089113 7517 heartbeater.cc:464] Registering TS with master...
I20260929 03:21:25.089325 7517 heartbeater.cc:511] Master 127.20.178.126:40469 requested a full tablet report, sending...
I20260929 03:21:25.089854 7329 ts_manager.cc:194] Registered new tserver with Master: 199fe8c8fad84ac298b6c12756afd961 (127.20.178.65:39185)
I20260929 03:21:25.090462 7329 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.178.65:46127
I20260929 03:21:25.092351 21193 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuoaHbX/build/release/bin/kudu as pid 7389
I20260929 03:21:25.092437 21193 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790651852476208-21193-0/minicluster-data/ts-0/wal/instance
I20260929 03:21:25.092581 21193 external_mini_cluster.cc:1428] Setting key 2234e4bb6eb847849fc52f1d0b209aa0
I20260929 03:21:25.093428 21193 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuoaHbX/build/release/bin/kudu
/tmp/dist-test-taskuoaHbX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790651852476208-21193-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790651852476208-21193-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790651852476208-21193-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790651852476208-21193-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.178.66:0
--local_ip_for_outbound_sockets=127.20.178.66
--webserver_interface=127.20.178.66
--webserver_port=0
--tserver_master_addrs=127.20.178.126:40469
--builtin_ntp_servers=127.20.178.84:41283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 03:21:25.172981 7520 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:21:25.173188 7520 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:21:25.173211 7520 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:21:25.174278 7520 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:21:25.174311 7520 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:21:25.174335 7520 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.178.66
I20260929 03:21:25.175573 7520 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.178.84:41283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790651852476208-21193-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790651852476208-21193-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.178.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790651852476208-21193-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.178.66
--webserver_port=0
--tserver_master_addrs=127.20.178.126:40469
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7520
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.178.66
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790651852476208-21193-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:21:25.175848 7520 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:21:25.176105 7520 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:21:25.181025 7526 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:21:25.181056 7525 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:21:25.181025 7528 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:21:25.181238 7520 server_base.cc:1034] running on GCE node
I20260929 03:21:25.181474 7520 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:21:25.181656 7520 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:21:25.182826 7520 hybrid_clock.cc:648] HybridClock initialized: now 1790652085182806 us; error 28 us; skew 500 ppm
I20260929 03:21:25.184039 7520 webserver.cc:460] Webserver started at http://127.20.178.66:43661/ using document root <none> and password file <none>
I20260929 03:21:25.184276 7520 fs_manager.cc:359] Metadata directory not provided
I20260929 03:21:25.184346 7520 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:21:25.184497 7520 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:21:25.185634 7520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790651852476208-21193-0/minicluster-data/ts-1/data/instance:
uuid: "9aa7be32cbc54727b68b1d9c3ce1ab99"
format_stamp: "Formatted at 2026-09-29 03:21:25 on dist-test-slave-c9qh"
server_key: "5c848c48cafe35ea4f4dd09b1d7286c2"
server_key_iv: "5d9cc383bdba6192315624d435fa820f"
server_key_version: "encryptionkey@0"
I20260929 03:21:25.186003 7520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790651852476208-21193-0/minicluster-data/ts-1/wal/instance:
uuid: "9aa7be32cbc54727b68b1d9c3ce1ab99"
format_stamp: "Formatted at 2026-09-29 03:21:25 on dist-test-slave-c9qh"
server_key: "5c848c48cafe35ea4f4dd09b1d7286c2"
server_key_iv: "5d9cc383bdba6192315624d435fa820f"
server_key_version: "encryptionkey@0"
I20260929 03:21:25.187280 7520 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 03:21:25.188002 7534 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:21:25.188171 7520 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 03:21:25.188452 7520 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790651852476208-21193-0/minicluster-data/ts-1/data,/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790651852476208-21193-0/minicluster-data/ts-1/wal
uuid: "9aa7be32cbc54727b68b1d9c3ce1ab99"
format_stamp: "Formatted at 2026-09-29 03:21:25 on dist-test-slave-c9qh"
server_key: "5c848c48cafe35ea4f4dd09b1d7286c2"
server_key_iv: "5d9cc383bdba6192315624d435fa820f"
server_key_version: "encryptionkey@0"
I20260929 03:21:25.188556 7520 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790651852476208-21193-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790651852476208-21193-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790651852476208-21193-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:21:25.206310 7520 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:21:25.206606 7520 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:21:25.206769 7520 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:21:25.206995 7520 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:21:25.207332 7520 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:21:25.207394 7520 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:21:25.207434 7520 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:21:25.207463 7520 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:21:25.213822 7520 rpc_server.cc:307] RPC server started. Bound to: 127.20.178.66:35285
I20260929 03:21:25.213865 7647 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.178.66:35285 every 8 connection(s)
I20260929 03:21:25.214185 7520 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790651852476208-21193-0/minicluster-data/ts-1/data/info.pb
I20260929 03:21:25.217926 21193 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuoaHbX/build/release/bin/kudu as pid 7520
I20260929 03:21:25.218053 21193 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790651852476208-21193-0/minicluster-data/ts-1/wal/instance
I20260929 03:21:25.218255 21193 external_mini_cluster.cc:1428] Setting key 76aea662e0d41fc06567fab13758ace8
I20260929 03:21:25.219256 21193 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuoaHbX/build/release/bin/kudu
/tmp/dist-test-taskuoaHbX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790651852476208-21193-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790651852476208-21193-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790651852476208-21193-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790651852476208-21193-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.178.67:0
--local_ip_for_outbound_sockets=127.20.178.67
--webserver_interface=127.20.178.67
--webserver_port=0
--tserver_master_addrs=127.20.178.126:40469
--builtin_ntp_servers=127.20.178.84:41283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 03:21:25.219726 7368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:25.214924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.66:34085 (local address 127.20.178.126:40469)
0929 03:21:25.214993 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0929 03:21:25.214995 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:21:25.215280 (+ 285us) server_negotiation.cc:407] Connection header received
0929 03:21:25.215762 (+ 482us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:21:25.215763 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:21:25.215774 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:21:25.215820 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:21:25.216529 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:25.217071 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:25.218024 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:25.218306 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:25.218393 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:21:25.219451 (+ 1058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:21:25.219452 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:21:25.219453 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:21:25.219458 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:21:25.219475 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:21:25.219480 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:21:25.219538 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0929 03:21:25.219615 (+ 77us) server_negotiation.cc:299] Negotiation successful
0929 03:21:25.219642 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 03:21:25.219964 7650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:25.214928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:40469 (local address 127.20.178.66:34085)
0929 03:21:25.215161 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0929 03:21:25.215206 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 03:21:25.215669 (+ 463us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:21:25.215881 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:21:25.215884 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:21:25.215892 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:21:25.216451 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:21:25.216456 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:21:25.217153 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:21:25.217154 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:21:25.217928 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:21:25.217930 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:21:25.218408 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:21:25.218409 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:21:25.218575 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:21:25.219244 (+ 669us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 03:21:25.219244 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:21:25.219281 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 03:21:25.219305 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:21:25.219541 (+ 236us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:21:25.219547 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:21:25.219550 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 03:21:25.219616 (+ 66us) client_negotiation.cc:240] Negotiation successful
0929 03:21:25.219713 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":87,"threads_started":1}
I20260929 03:21:25.220485 7648 heartbeater.cc:347] Connected to a master server at 127.20.178.126:40469
I20260929 03:21:25.220590 7648 heartbeater.cc:464] Registering TS with master...
I20260929 03:21:25.220793 7648 heartbeater.cc:511] Master 127.20.178.126:40469 requested a full tablet report, sending...
I20260929 03:21:25.221179 7329 ts_manager.cc:194] Registered new tserver with Master: 9aa7be32cbc54727b68b1d9c3ce1ab99 (127.20.178.66:35285)
I20260929 03:21:25.221560 7329 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.178.66:34085
W20260929 03:21:25.299887 7651 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:21:25.300083 7651 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:21:25.300104 7651 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:21:25.301316 7651 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:21:25.301391 7651 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:21:25.301434 7651 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.178.67
I20260929 03:21:25.303120 7651 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.178.84:41283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790651852476208-21193-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790651852476208-21193-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.178.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790651852476208-21193-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.178.67
--webserver_port=0
--tserver_master_addrs=127.20.178.126:40469
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7651
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.178.67
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790651852476208-21193-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:21:25.303421 7651 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:21:25.303774 7651 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:21:25.308990 7656 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:21:25.309048 7651 server_base.cc:1034] running on GCE node
W20260929 03:21:25.309036 7659 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:21:25.309001 7657 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:21:25.309528 7651 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:21:25.309793 7651 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:21:25.310966 7651 hybrid_clock.cc:648] HybridClock initialized: now 1790652085310949 us; error 28 us; skew 500 ppm
I20260929 03:21:25.312335 7651 webserver.cc:460] Webserver started at http://127.20.178.67:42135/ using document root <none> and password file <none>
I20260929 03:21:25.312592 7651 fs_manager.cc:359] Metadata directory not provided
I20260929 03:21:25.312690 7651 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:21:25.312860 7651 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:21:25.313930 7651 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790651852476208-21193-0/minicluster-data/ts-2/data/instance:
uuid: "775fc93dfa254be2a79f1630fb893bc7"
format_stamp: "Formatted at 2026-09-29 03:21:25 on dist-test-slave-c9qh"
server_key: "e11ab5f1d32ead7851cfcba80d26ced4"
server_key_iv: "146a5e2345111577e3ddc229b9aa1491"
server_key_version: "encryptionkey@0"
I20260929 03:21:25.314344 7651 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790651852476208-21193-0/minicluster-data/ts-2/wal/instance:
uuid: "775fc93dfa254be2a79f1630fb893bc7"
format_stamp: "Formatted at 2026-09-29 03:21:25 on dist-test-slave-c9qh"
server_key: "e11ab5f1d32ead7851cfcba80d26ced4"
server_key_iv: "146a5e2345111577e3ddc229b9aa1491"
server_key_version: "encryptionkey@0"
I20260929 03:21:25.315690 7651 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 03:21:25.316383 7665 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:21:25.316555 7651 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 03:21:25.316666 7651 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790651852476208-21193-0/minicluster-data/ts-2/data,/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790651852476208-21193-0/minicluster-data/ts-2/wal
uuid: "775fc93dfa254be2a79f1630fb893bc7"
format_stamp: "Formatted at 2026-09-29 03:21:25 on dist-test-slave-c9qh"
server_key: "e11ab5f1d32ead7851cfcba80d26ced4"
server_key_iv: "146a5e2345111577e3ddc229b9aa1491"
server_key_version: "encryptionkey@0"
I20260929 03:21:25.316766 7651 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790651852476208-21193-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790651852476208-21193-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790651852476208-21193-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:21:25.352166 7651 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:21:25.352509 7651 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:21:25.352705 7651 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:21:25.352945 7651 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:21:25.353300 7651 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:21:25.353356 7651 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:21:25.353397 7651 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:21:25.353425 7651 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:21:25.358978 7651 rpc_server.cc:307] RPC server started. Bound to: 127.20.178.67:44161
I20260929 03:21:25.359027 7778 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.178.67:44161 every 8 connection(s)
I20260929 03:21:25.359354 7651 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790651852476208-21193-0/minicluster-data/ts-2/data/info.pb
I20260929 03:21:25.364599 21193 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuoaHbX/build/release/bin/kudu as pid 7651
I20260929 03:21:25.364727 21193 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790651852476208-21193-0/minicluster-data/ts-2/wal/instance
I20260929 03:21:25.364869 7368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:25.360051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.67:53699 (local address 127.20.178.126:40469)
0929 03:21:25.360122 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0929 03:21:25.360124 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:21:25.360399 (+ 275us) server_negotiation.cc:407] Connection header received
0929 03:21:25.360957 (+ 558us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:21:25.360959 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:21:25.360970 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:21:25.361025 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:21:25.361662 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:25.362153 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:25.363147 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:25.363343 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:25.363407 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:21:25.364393 (+ 986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:21:25.364395 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:21:25.364396 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:21:25.364401 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:21:25.364423 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:21:25.364428 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:21:25.364489 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0929 03:21:25.364616 (+ 127us) server_negotiation.cc:299] Negotiation successful
0929 03:21:25.364683 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260929 03:21:25.364953 21193 external_mini_cluster.cc:1428] Setting key cb309fdbf90487527be5e182270ce4fe
I20260929 03:21:25.365028 7781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:25.360056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:40469 (local address 127.20.178.67:53699)
0929 03:21:25.360269 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0929 03:21:25.360303 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0929 03:21:25.360863 (+ 560us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:21:25.361078 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:21:25.361082 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:21:25.361090 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:21:25.361588 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:21:25.361592 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:21:25.362253 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:21:25.362254 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:21:25.363049 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:21:25.363051 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:21:25.363423 (+ 372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:21:25.363424 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:21:25.363530 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:21:25.364194 (+ 664us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 03:21:25.364195 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:21:25.364256 (+ 61us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 03:21:25.364284 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:21:25.364522 (+ 238us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:21:25.364530 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:21:25.364533 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 03:21:25.364613 (+ 80us) client_negotiation.cc:240] Negotiation successful
0929 03:21:25.364756 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":89,"threads_started":1}
I20260929 03:21:25.365546 7779 heartbeater.cc:347] Connected to a master server at 127.20.178.126:40469
I20260929 03:21:25.365664 7779 heartbeater.cc:464] Registering TS with master...
I20260929 03:21:25.365869 7779 heartbeater.cc:511] Master 127.20.178.126:40469 requested a full tablet report, sending...
I20260929 03:21:25.366257 7329 ts_manager.cc:194] Registered new tserver with Master: 775fc93dfa254be2a79f1630fb893bc7 (127.20.178.67:44161)
I20260929 03:21:25.366647 7329 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.178.67:53699
I20260929 03:21:25.376856 21193 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 03:21:25.381274 7368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:25.378222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46872 (local address 127.20.178.126:40469)
0929 03:21:25.378318 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0929 03:21:25.378320 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:21:25.378425 (+ 105us) server_negotiation.cc:407] Connection header received
0929 03:21:25.378501 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:21:25.378502 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:21:25.378513 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:21:25.378557 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:21:25.378744 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:25.379239 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:25.379979 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:25.380192 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:25.380270 (+ 78us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:21:25.380968 (+ 698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:21:25.380969 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:21:25.380969 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:21:25.380974 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:21:25.380987 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:21:25.380991 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:21:25.381105 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0929 03:21:25.381186 (+ 81us) server_negotiation.cc:299] Negotiation successful
0929 03:21:25.381197 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 03:21:25.382810 7329 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46872:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 03:21:25.384471 7329 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 03:21:25.390760 7788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:25.386596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.67:44161 (local address 127.0.0.1:60848)
0929 03:21:25.386769 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0929 03:21:25.386806 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 03:21:25.386873 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:21:25.387217 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:21:25.387221 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:21:25.387223 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:21:25.387284 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:21:25.387286 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:21:25.388395 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:21:25.388395 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:21:25.389831 (+ 1436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:21:25.389834 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:21:25.390525 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:21:25.390526 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:21:25.390571 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:21:25.390574 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 03:21:25.390603 (+ 29us) client_negotiation.cc:240] Negotiation successful
0929 03:21:25.390662 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"mutex_wait_us":3,"thread_start_us":87,"threads_started":1}
I20260929 03:21:25.390760 7787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:25.386599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.66:35285 (local address 127.0.0.1:49596)
0929 03:21:25.386854 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0929 03:21:25.386864 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 03:21:25.386905 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:21:25.387306 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:21:25.387307 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:21:25.387309 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:21:25.387354 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:21:25.387356 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:21:25.388683 (+ 1327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:21:25.388684 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:21:25.389515 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:21:25.389518 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:21:25.390452 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:21:25.390453 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:21:25.390537 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:21:25.390551 (+ 14us) client_negotiation.cc:769] Sending connection context
0929 03:21:25.390626 (+ 75us) client_negotiation.cc:240] Negotiation successful
0929 03:21:25.390662 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"mutex_wait_us":53,"thread_start_us":59,"threads_started":1}
I20260929 03:21:25.391004 7789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:25.386622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.65:39185 (local address 127.0.0.1:42956)
0929 03:21:25.386981 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0929 03:21:25.386988 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0929 03:21:25.387023 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:21:25.387381 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:21:25.387382 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:21:25.387384 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:21:25.387426 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:21:25.387428 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:21:25.388740 (+ 1312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:21:25.388741 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:21:25.389863 (+ 1122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:21:25.389865 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:21:25.390775 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:21:25.390776 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:21:25.390842 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:21:25.390847 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 03:21:25.390882 (+ 35us) client_negotiation.cc:240] Negotiation successful
0929 03:21:25.390929 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":347,"mutex_wait_us":19,"thread_start_us":65,"threads_started":1}
I20260929 03:21:25.391161 7791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:25.386822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60848 (local address 127.20.178.67:44161)
0929 03:21:25.387004 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0929 03:21:25.387009 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:21:25.387021 (+ 12us) server_negotiation.cc:407] Connection header received
0929 03:21:25.387061 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:21:25.387063 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:21:25.387092 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:21:25.387208 (+ 116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:21:25.387367 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:25.388305 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:25.389923 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:25.390430 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:25.390884 (+ 454us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:21:25.390991 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0929 03:21:25.391017 (+ 26us) server_negotiation.cc:299] Negotiation successful
0929 03:21:25.391058 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":112,"threads_started":1}
I20260929 03:21:25.390792 7790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:25.386674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49596 (local address 127.20.178.66:35285)
0929 03:21:25.387150 (+ 476us) server_negotiation.cc:206] Beginning negotiation
0929 03:21:25.387155 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:21:25.387166 (+ 11us) server_negotiation.cc:407] Connection header received
0929 03:21:25.387204 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:21:25.387206 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:21:25.387220 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:21:25.387312 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:21:25.387415 (+ 103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:25.388548 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:25.389636 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:25.390355 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:25.390430 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:21:25.390514 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0929 03:21:25.390644 (+ 130us) server_negotiation.cc:299] Negotiation successful
0929 03:21:25.390682 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":426,"thread_start_us":98,"threads_started":1}
I20260929 03:21:25.391435 7792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:25.387028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42956 (local address 127.20.178.65:39185)
0929 03:21:25.387224 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0929 03:21:25.387228 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:21:25.387237 (+ 9us) server_negotiation.cc:407] Connection header received
0929 03:21:25.387272 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:21:25.387274 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:21:25.387289 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:21:25.387496 (+ 207us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:21:25.387555 (+ 59us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:25.388609 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:25.389942 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:25.390668 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:25.391166 (+ 498us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:21:25.391280 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0929 03:21:25.391311 (+ 31us) server_negotiation.cc:299] Negotiation successful
0929 03:21:25.391357 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":98,"threads_started":1}
I20260929 03:21:25.391911 7582 tablet_service.cc:1467] Processing CreateTablet for tablet 794e93a3a8464779a038eab8db5e6b68 (DEFAULT_TABLE table=test-table [id=ec8540527d2d479dbd96a2db9df9d8bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:21:25.392009 7451 tablet_service.cc:1467] Processing CreateTablet for tablet 794e93a3a8464779a038eab8db5e6b68 (DEFAULT_TABLE table=test-table [id=ec8540527d2d479dbd96a2db9df9d8bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:21:25.392274 7451 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 794e93a3a8464779a038eab8db5e6b68. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:21:25.392282 7582 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 794e93a3a8464779a038eab8db5e6b68. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:21:25.393029 7713 tablet_service.cc:1467] Processing CreateTablet for tablet 794e93a3a8464779a038eab8db5e6b68 (DEFAULT_TABLE table=test-table [id=ec8540527d2d479dbd96a2db9df9d8bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:21:25.393347 7713 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 794e93a3a8464779a038eab8db5e6b68. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:21:25.395035 7793 tablet_bootstrap.cc:492] T 794e93a3a8464779a038eab8db5e6b68 P 199fe8c8fad84ac298b6c12756afd961: Bootstrap starting.
I20260929 03:21:25.395604 7794 tablet_bootstrap.cc:492] T 794e93a3a8464779a038eab8db5e6b68 P 9aa7be32cbc54727b68b1d9c3ce1ab99: Bootstrap starting.
I20260929 03:21:25.395977 7793 tablet_bootstrap.cc:654] T 794e93a3a8464779a038eab8db5e6b68 P 199fe8c8fad84ac298b6c12756afd961: Neither blocks nor log segments found. Creating new log.
I20260929 03:21:25.396255 7794 tablet_bootstrap.cc:654] T 794e93a3a8464779a038eab8db5e6b68 P 9aa7be32cbc54727b68b1d9c3ce1ab99: Neither blocks nor log segments found. Creating new log.
I20260929 03:21:25.396281 7793 log.cc:824] T 794e93a3a8464779a038eab8db5e6b68 P 199fe8c8fad84ac298b6c12756afd961: Log is configured to *not* fsync() on all Append() calls
I20260929 03:21:25.396420 7795 tablet_bootstrap.cc:492] T 794e93a3a8464779a038eab8db5e6b68 P 775fc93dfa254be2a79f1630fb893bc7: Bootstrap starting.
I20260929 03:21:25.396546 7794 log.cc:824] T 794e93a3a8464779a038eab8db5e6b68 P 9aa7be32cbc54727b68b1d9c3ce1ab99: Log is configured to *not* fsync() on all Append() calls
I20260929 03:21:25.397047 7793 tablet_bootstrap.cc:492] T 794e93a3a8464779a038eab8db5e6b68 P 199fe8c8fad84ac298b6c12756afd961: No bootstrap required, opened a new log
I20260929 03:21:25.397070 7795 tablet_bootstrap.cc:654] T 794e93a3a8464779a038eab8db5e6b68 P 775fc93dfa254be2a79f1630fb893bc7: Neither blocks nor log segments found. Creating new log.
I20260929 03:21:25.397122 7793 ts_tablet_manager.cc:1397] T 794e93a3a8464779a038eab8db5e6b68 P 199fe8c8fad84ac298b6c12756afd961: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 03:21:25.397359 7794 tablet_bootstrap.cc:492] T 794e93a3a8464779a038eab8db5e6b68 P 9aa7be32cbc54727b68b1d9c3ce1ab99: No bootstrap required, opened a new log
I20260929 03:21:25.397382 7795 log.cc:824] T 794e93a3a8464779a038eab8db5e6b68 P 775fc93dfa254be2a79f1630fb893bc7: Log is configured to *not* fsync() on all Append() calls
I20260929 03:21:25.397426 7794 ts_tablet_manager.cc:1397] T 794e93a3a8464779a038eab8db5e6b68 P 9aa7be32cbc54727b68b1d9c3ce1ab99: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 03:21:25.398015 7795 tablet_bootstrap.cc:492] T 794e93a3a8464779a038eab8db5e6b68 P 775fc93dfa254be2a79f1630fb893bc7: No bootstrap required, opened a new log
I20260929 03:21:25.398093 7795 ts_tablet_manager.cc:1397] T 794e93a3a8464779a038eab8db5e6b68 P 775fc93dfa254be2a79f1630fb893bc7: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260929 03:21:25.399787 7793 raft_consensus.cc:348] T 794e93a3a8464779a038eab8db5e6b68 P 199fe8c8fad84ac298b6c12756afd961 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "775fc93dfa254be2a79f1630fb893bc7" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 44161 } } peers { permanent_uuid: "9aa7be32cbc54727b68b1d9c3ce1ab99" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 35285 } } peers { permanent_uuid: "199fe8c8fad84ac298b6c12756afd961" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 39185 } }
I20260929 03:21:25.399959 7793 raft_consensus.cc:374] T 794e93a3a8464779a038eab8db5e6b68 P 199fe8c8fad84ac298b6c12756afd961 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:21:25.399993 7793 raft_consensus.cc:729] T 794e93a3a8464779a038eab8db5e6b68 P 199fe8c8fad84ac298b6c12756afd961 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 199fe8c8fad84ac298b6c12756afd961, State: Initialized, Role: FOLLOWER
I20260929 03:21:25.399906 7794 raft_consensus.cc:348] T 794e93a3a8464779a038eab8db5e6b68 P 9aa7be32cbc54727b68b1d9c3ce1ab99 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "775fc93dfa254be2a79f1630fb893bc7" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 44161 } } peers { permanent_uuid: "9aa7be32cbc54727b68b1d9c3ce1ab99" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 35285 } } peers { permanent_uuid: "199fe8c8fad84ac298b6c12756afd961" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 39185 } }
I20260929 03:21:25.400035 7794 raft_consensus.cc:374] T 794e93a3a8464779a038eab8db5e6b68 P 9aa7be32cbc54727b68b1d9c3ce1ab99 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:21:25.400063 7794 raft_consensus.cc:729] T 794e93a3a8464779a038eab8db5e6b68 P 9aa7be32cbc54727b68b1d9c3ce1ab99 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9aa7be32cbc54727b68b1d9c3ce1ab99, State: Initialized, Role: FOLLOWER
I20260929 03:21:25.399941 7795 raft_consensus.cc:348] T 794e93a3a8464779a038eab8db5e6b68 P 775fc93dfa254be2a79f1630fb893bc7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "775fc93dfa254be2a79f1630fb893bc7" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 44161 } } peers { permanent_uuid: "9aa7be32cbc54727b68b1d9c3ce1ab99" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 35285 } } peers { permanent_uuid: "199fe8c8fad84ac298b6c12756afd961" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 39185 } }
I20260929 03:21:25.400130 7795 raft_consensus.cc:374] T 794e93a3a8464779a038eab8db5e6b68 P 775fc93dfa254be2a79f1630fb893bc7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:21:25.400170 7795 raft_consensus.cc:729] T 794e93a3a8464779a038eab8db5e6b68 P 775fc93dfa254be2a79f1630fb893bc7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 775fc93dfa254be2a79f1630fb893bc7, State: Initialized, Role: FOLLOWER
I20260929 03:21:25.400184 7794 consensus_queue.cc:260] T 794e93a3a8464779a038eab8db5e6b68 P 9aa7be32cbc54727b68b1d9c3ce1ab99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "775fc93dfa254be2a79f1630fb893bc7" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 44161 } } peers { permanent_uuid: "9aa7be32cbc54727b68b1d9c3ce1ab99" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 35285 } } peers { permanent_uuid: "199fe8c8fad84ac298b6c12756afd961" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 39185 } }
I20260929 03:21:25.400272 7793 consensus_queue.cc:260] T 794e93a3a8464779a038eab8db5e6b68 P 199fe8c8fad84ac298b6c12756afd961 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "775fc93dfa254be2a79f1630fb893bc7" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 44161 } } peers { permanent_uuid: "9aa7be32cbc54727b68b1d9c3ce1ab99" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 35285 } } peers { permanent_uuid: "199fe8c8fad84ac298b6c12756afd961" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 39185 } }
I20260929 03:21:25.400287 7795 consensus_queue.cc:260] T 794e93a3a8464779a038eab8db5e6b68 P 775fc93dfa254be2a79f1630fb893bc7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "775fc93dfa254be2a79f1630fb893bc7" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 44161 } } peers { permanent_uuid: "9aa7be32cbc54727b68b1d9c3ce1ab99" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 35285 } } peers { permanent_uuid: "199fe8c8fad84ac298b6c12756afd961" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 39185 } }
I20260929 03:21:25.400460 7794 ts_tablet_manager.cc:1428] T 794e93a3a8464779a038eab8db5e6b68 P 9aa7be32cbc54727b68b1d9c3ce1ab99: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260929 03:21:25.400508 7793 ts_tablet_manager.cc:1428] T 794e93a3a8464779a038eab8db5e6b68 P 199fe8c8fad84ac298b6c12756afd961: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260929 03:21:25.400548 7795 ts_tablet_manager.cc:1428] T 794e93a3a8464779a038eab8db5e6b68 P 775fc93dfa254be2a79f1630fb893bc7: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260929 03:21:25.400581 7648 heartbeater.cc:503] Master 127.20.178.126:40469 was elected leader, sending a full tablet report...
I20260929 03:21:25.400696 7779 heartbeater.cc:503] Master 127.20.178.126:40469 was elected leader, sending a full tablet report...
I20260929 03:21:25.400898 7517 heartbeater.cc:503] Master 127.20.178.126:40469 was elected leader, sending a full tablet report...
I20260929 03:21:25.429445 7800 raft_consensus.cc:482] T 794e93a3a8464779a038eab8db5e6b68 P 199fe8c8fad84ac298b6c12756afd961 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:21:25.429561 7800 raft_consensus.cc:504] T 794e93a3a8464779a038eab8db5e6b68 P 199fe8c8fad84ac298b6c12756afd961 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "775fc93dfa254be2a79f1630fb893bc7" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 44161 } } peers { permanent_uuid: "9aa7be32cbc54727b68b1d9c3ce1ab99" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 35285 } } peers { permanent_uuid: "199fe8c8fad84ac298b6c12756afd961" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 39185 } }
I20260929 03:21:25.429863 7800 leader_election.cc:288] T 794e93a3a8464779a038eab8db5e6b68 P 199fe8c8fad84ac298b6c12756afd961 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 775fc93dfa254be2a79f1630fb893bc7 (127.20.178.67:44161), 9aa7be32cbc54727b68b1d9c3ce1ab99 (127.20.178.66:35285)
I20260929 03:21:25.433607 7790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:25.430123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.65:35671 (local address 127.20.178.66:35285)
0929 03:21:25.430170 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0929 03:21:25.430172 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:21:25.430354 (+ 182us) server_negotiation.cc:407] Connection header received
0929 03:21:25.430382 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:21:25.430383 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:21:25.430393 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:21:25.430472 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:21:25.430611 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:25.431297 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:25.432684 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:25.433232 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:25.433311 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:21:25.433318 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0929 03:21:25.433495 (+ 177us) server_negotiation.cc:299] Negotiation successful
0929 03:21:25.433550 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260929 03:21:25.433607 7791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:25.430117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.65:43581 (local address 127.20.178.67:44161)
0929 03:21:25.430161 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0929 03:21:25.430163 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:21:25.430176 (+ 13us) server_negotiation.cc:407] Connection header received
0929 03:21:25.430205 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:21:25.430205 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:21:25.430221 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:21:25.430277 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:21:25.430447 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:25.431199 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:25.432668 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:25.433232 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:25.433311 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:21:25.433318 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0929 03:21:25.433495 (+ 177us) server_negotiation.cc:299] Negotiation successful
0929 03:21:25.433536 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260929 03:21:25.433631 7519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:25.430033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.67:44161 (local address 127.20.178.65:43581)
0929 03:21:25.430083 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0929 03:21:25.430095 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 03:21:25.430136 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:21:25.430287 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:21:25.430288 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:21:25.430290 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:21:25.430366 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:21:25.430367 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:21:25.431293 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:21:25.431294 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:21:25.432560 (+ 1266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:21:25.432563 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:21:25.433327 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:21:25.433328 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:21:25.433410 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:21:25.433416 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 03:21:25.433479 (+ 63us) client_negotiation.cc:240] Negotiation successful
0929 03:21:25.433527 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32,"mutex_wait_us":10}
I20260929 03:21:25.433857 7802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:25.430043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.66:35285 (local address 127.20.178.65:35671)
0929 03:21:25.430277 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0929 03:21:25.430286 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 03:21:25.430340 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:21:25.430487 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:21:25.430488 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:21:25.430489 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:21:25.430543 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:21:25.430545 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:21:25.431383 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:21:25.431383 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:21:25.432560 (+ 1177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:21:25.432563 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:21:25.433327 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:21:25.433328 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:21:25.433410 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:21:25.433417 (+ 7us) client_negotiation.cc:769] Sending connection context
0929 03:21:25.433479 (+ 62us) client_negotiation.cc:240] Negotiation successful
0929 03:21:25.433526 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216,"mutex_wait_us":63,"thread_start_us":99,"threads_started":1}
I20260929 03:21:25.433975 7733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "794e93a3a8464779a038eab8db5e6b68" candidate_uuid: "199fe8c8fad84ac298b6c12756afd961" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "775fc93dfa254be2a79f1630fb893bc7" is_pre_election: true
I20260929 03:21:25.434132 7733 raft_consensus.cc:2454] T 794e93a3a8464779a038eab8db5e6b68 P 775fc93dfa254be2a79f1630fb893bc7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 199fe8c8fad84ac298b6c12756afd961 in term 0.
I20260929 03:21:25.434221 7602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "794e93a3a8464779a038eab8db5e6b68" candidate_uuid: "199fe8c8fad84ac298b6c12756afd961" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9aa7be32cbc54727b68b1d9c3ce1ab99" is_pre_election: true
I20260929 03:21:25.434392 7602 raft_consensus.cc:2454] T 794e93a3a8464779a038eab8db5e6b68 P 9aa7be32cbc54727b68b1d9c3ce1ab99 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 199fe8c8fad84ac298b6c12756afd961 in term 0.
I20260929 03:21:25.434396 7404 leader_election.cc:302] T 794e93a3a8464779a038eab8db5e6b68 P 199fe8c8fad84ac298b6c12756afd961 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 199fe8c8fad84ac298b6c12756afd961, 775fc93dfa254be2a79f1630fb893bc7; no voters:
I20260929 03:21:25.434543 7800 raft_consensus.cc:2781] T 794e93a3a8464779a038eab8db5e6b68 P 199fe8c8fad84ac298b6c12756afd961 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 03:21:25.434612 7800 raft_consensus.cc:482] T 794e93a3a8464779a038eab8db5e6b68 P 199fe8c8fad84ac298b6c12756afd961 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 03:21:25.434644 7800 raft_consensus.cc:3037] T 794e93a3a8464779a038eab8db5e6b68 P 199fe8c8fad84ac298b6c12756afd961 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:21:25.435343 7800 raft_consensus.cc:504] T 794e93a3a8464779a038eab8db5e6b68 P 199fe8c8fad84ac298b6c12756afd961 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "775fc93dfa254be2a79f1630fb893bc7" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 44161 } } peers { permanent_uuid: "9aa7be32cbc54727b68b1d9c3ce1ab99" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 35285 } } peers { permanent_uuid: "199fe8c8fad84ac298b6c12756afd961" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 39185 } }
I20260929 03:21:25.435489 7800 leader_election.cc:288] T 794e93a3a8464779a038eab8db5e6b68 P 199fe8c8fad84ac298b6c12756afd961 [CANDIDATE]: Term 1 election: Requested vote from peers 775fc93dfa254be2a79f1630fb893bc7 (127.20.178.67:44161), 9aa7be32cbc54727b68b1d9c3ce1ab99 (127.20.178.66:35285)
I20260929 03:21:25.435629 7733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "794e93a3a8464779a038eab8db5e6b68" candidate_uuid: "199fe8c8fad84ac298b6c12756afd961" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "775fc93dfa254be2a79f1630fb893bc7"
I20260929 03:21:25.435672 7602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "794e93a3a8464779a038eab8db5e6b68" candidate_uuid: "199fe8c8fad84ac298b6c12756afd961" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9aa7be32cbc54727b68b1d9c3ce1ab99"
I20260929 03:21:25.435716 7733 raft_consensus.cc:3037] T 794e93a3a8464779a038eab8db5e6b68 P 775fc93dfa254be2a79f1630fb893bc7 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:21:25.435737 7602 raft_consensus.cc:3037] T 794e93a3a8464779a038eab8db5e6b68 P 9aa7be32cbc54727b68b1d9c3ce1ab99 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:21:25.436606 7733 raft_consensus.cc:2454] T 794e93a3a8464779a038eab8db5e6b68 P 775fc93dfa254be2a79f1630fb893bc7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 199fe8c8fad84ac298b6c12756afd961 in term 1.
I20260929 03:21:25.436594 7602 raft_consensus.cc:2454] T 794e93a3a8464779a038eab8db5e6b68 P 9aa7be32cbc54727b68b1d9c3ce1ab99 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 199fe8c8fad84ac298b6c12756afd961 in term 1.
I20260929 03:21:25.436857 7405 leader_election.cc:302] T 794e93a3a8464779a038eab8db5e6b68 P 199fe8c8fad84ac298b6c12756afd961 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 199fe8c8fad84ac298b6c12756afd961, 9aa7be32cbc54727b68b1d9c3ce1ab99; no voters:
I20260929 03:21:25.436955 7800 raft_consensus.cc:2781] T 794e93a3a8464779a038eab8db5e6b68 P 199fe8c8fad84ac298b6c12756afd961 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:21:25.437093 7800 raft_consensus.cc:686] T 794e93a3a8464779a038eab8db5e6b68 P 199fe8c8fad84ac298b6c12756afd961 [term 1 LEADER]: Becoming Leader. State: Replica: 199fe8c8fad84ac298b6c12756afd961, State: Running, Role: LEADER
I20260929 03:21:25.437192 7800 consensus_queue.cc:237] T 794e93a3a8464779a038eab8db5e6b68 P 199fe8c8fad84ac298b6c12756afd961 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "775fc93dfa254be2a79f1630fb893bc7" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 44161 } } peers { permanent_uuid: "9aa7be32cbc54727b68b1d9c3ce1ab99" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 35285 } } peers { permanent_uuid: "199fe8c8fad84ac298b6c12756afd961" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 39185 } }
I20260929 03:21:25.437973 7328 catalog_manager.cc:5495] T 794e93a3a8464779a038eab8db5e6b68 P 199fe8c8fad84ac298b6c12756afd961 reported cstate change: term changed from 0 to 1, leader changed from <none> to 199fe8c8fad84ac298b6c12756afd961 (127.20.178.65). New cstate: current_term: 1 leader_uuid: "199fe8c8fad84ac298b6c12756afd961" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "775fc93dfa254be2a79f1630fb893bc7" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 44161 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9aa7be32cbc54727b68b1d9c3ce1ab99" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 35285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "199fe8c8fad84ac298b6c12756afd961" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 39185 } health_report { overall_health: HEALTHY } } }
I20260929 03:21:25.450043 7792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:25.447730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42960 (local address 127.20.178.65:39185)
0929 03:21:25.447793 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0929 03:21:25.447795 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:21:25.447803 (+ 8us) server_negotiation.cc:407] Connection header received
0929 03:21:25.447830 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:21:25.447831 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:21:25.447851 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:21:25.447907 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:21:25.448058 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:25.448579 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:25.449392 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:25.449575 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:25.449630 (+ 55us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:21:25.449766 (+ 136us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:21:25.449849 (+ 83us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:21:25.449919 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0929 03:21:25.449961 (+ 42us) server_negotiation.cc:299] Negotiation successful
0929 03:21:25.449970 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260929 03:21:25.451864 7733 raft_consensus.cc:1261] T 794e93a3a8464779a038eab8db5e6b68 P 775fc93dfa254be2a79f1630fb893bc7 [term 1 FOLLOWER]: Refusing update from remote peer 199fe8c8fad84ac298b6c12756afd961: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:21:25.451864 7602 raft_consensus.cc:1261] T 794e93a3a8464779a038eab8db5e6b68 P 9aa7be32cbc54727b68b1d9c3ce1ab99 [term 1 FOLLOWER]: Refusing update from remote peer 199fe8c8fad84ac298b6c12756afd961: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:21:25.452257 7803 consensus_queue.cc:1035] T 794e93a3a8464779a038eab8db5e6b68 P 199fe8c8fad84ac298b6c12756afd961 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9aa7be32cbc54727b68b1d9c3ce1ab99" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 35285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:21:25.452409 7803 consensus_queue.cc:1035] T 794e93a3a8464779a038eab8db5e6b68 P 199fe8c8fad84ac298b6c12756afd961 [LEADER]: Connected to new peer: Peer: permanent_uuid: "775fc93dfa254be2a79f1630fb893bc7" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 44161 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:21:25.454959 7806 mvcc.cc:205] Tried to move back new op lower bound from 7334510942007955456 to 7334510941951229952. Current Snapshot: MvccSnapshot[applied={T|T < 7334510942007955456}]
I20260929 03:21:25.455950 7807 mvcc.cc:205] Tried to move back new op lower bound from 7334510942007955456 to 7334510941951229952. Current Snapshot: MvccSnapshot[applied={T|T < 7334510942007955456 or (T in {7334510942007955456})}]
I20260929 03:21:25.457307 7329 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46872:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:21:25.457398 7329 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46872:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:21:25.458117 7329 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ee6f31141ac54883b1348b436e10df2c: Sending DeleteTablet for 3 replicas of tablet 794e93a3a8464779a038eab8db5e6b68
I20260929 03:21:25.458446 7451 tablet_service.cc:1514] Processing DeleteTablet for tablet 794e93a3a8464779a038eab8db5e6b68 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:21:25 UTC) from {username='slave'} at 127.0.0.1:42956
I20260929 03:21:25.458464 7713 tablet_service.cc:1514] Processing DeleteTablet for tablet 794e93a3a8464779a038eab8db5e6b68 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:21:25 UTC) from {username='slave'} at 127.0.0.1:60848
I20260929 03:21:25.458451 7582 tablet_service.cc:1514] Processing DeleteTablet for tablet 794e93a3a8464779a038eab8db5e6b68 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:21:25 UTC) from {username='slave'} at 127.0.0.1:49596
I20260929 03:21:25.458736 7817 tablet_replica.cc:330] T 794e93a3a8464779a038eab8db5e6b68 P 199fe8c8fad84ac298b6c12756afd961: stopping tablet replica
I20260929 03:21:25.458786 7818 tablet_replica.cc:330] T 794e93a3a8464779a038eab8db5e6b68 P 9aa7be32cbc54727b68b1d9c3ce1ab99: stopping tablet replica
I20260929 03:21:25.458838 7817 raft_consensus.cc:2229] T 794e93a3a8464779a038eab8db5e6b68 P 199fe8c8fad84ac298b6c12756afd961 [term 1 LEADER]: Raft consensus shutting down.
I20260929 03:21:25.458859 7818 raft_consensus.cc:2229] T 794e93a3a8464779a038eab8db5e6b68 P 9aa7be32cbc54727b68b1d9c3ce1ab99 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:21:25.458918 7817 raft_consensus.cc:2258] T 794e93a3a8464779a038eab8db5e6b68 P 199fe8c8fad84ac298b6c12756afd961 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:21:25.459115 21193 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuoaHbX/build/release/bin/kudu with pid 7389
I20260929 03:21:25.459362 7817 ts_tablet_manager.cc:1905] T 794e93a3a8464779a038eab8db5e6b68 P 199fe8c8fad84ac298b6c12756afd961: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:21:25.459729 7816 tablet_replica.cc:330] T 794e93a3a8464779a038eab8db5e6b68 P 775fc93dfa254be2a79f1630fb893bc7: stopping tablet replica
I20260929 03:21:25.460098 7816 raft_consensus.cc:2229] T 794e93a3a8464779a038eab8db5e6b68 P 775fc93dfa254be2a79f1630fb893bc7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:21:25.460238 7816 raft_consensus.cc:2258] T 794e93a3a8464779a038eab8db5e6b68 P 775fc93dfa254be2a79f1630fb893bc7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:21:25.460667 7816 ts_tablet_manager.cc:1905] T 794e93a3a8464779a038eab8db5e6b68 P 775fc93dfa254be2a79f1630fb893bc7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:21:25.461359 7818 raft_consensus.cc:2258] T 794e93a3a8464779a038eab8db5e6b68 P 9aa7be32cbc54727b68b1d9c3ce1ab99 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:21:25.461748 7818 ts_tablet_manager.cc:1905] T 794e93a3a8464779a038eab8db5e6b68 P 9aa7be32cbc54727b68b1d9c3ce1ab99: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:21:25.462613 7816 ts_tablet_manager.cc:1918] T 794e93a3a8464779a038eab8db5e6b68 P 775fc93dfa254be2a79f1630fb893bc7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:21:25.462688 7816 log.cc:1196] T 794e93a3a8464779a038eab8db5e6b68 P 775fc93dfa254be2a79f1630fb893bc7: Deleting WAL directory at /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790651852476208-21193-0/minicluster-data/ts-2/wal/wals/794e93a3a8464779a038eab8db5e6b68
I20260929 03:21:25.462925 7816 ts_tablet_manager.cc:1939] T 794e93a3a8464779a038eab8db5e6b68 P 775fc93dfa254be2a79f1630fb893bc7: Deleting consensus metadata
I20260929 03:21:25.463152 7818 ts_tablet_manager.cc:1918] T 794e93a3a8464779a038eab8db5e6b68 P 9aa7be32cbc54727b68b1d9c3ce1ab99: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:21:25.463235 7818 log.cc:1196] T 794e93a3a8464779a038eab8db5e6b68 P 9aa7be32cbc54727b68b1d9c3ce1ab99: Deleting WAL directory at /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790651852476208-21193-0/minicluster-data/ts-1/wal/wals/794e93a3a8464779a038eab8db5e6b68
I20260929 03:21:25.463528 7818 ts_tablet_manager.cc:1939] T 794e93a3a8464779a038eab8db5e6b68 P 9aa7be32cbc54727b68b1d9c3ce1ab99: Deleting consensus metadata
I20260929 03:21:25.463508 7313 catalog_manager.cc:4841] TS 775fc93dfa254be2a79f1630fb893bc7 (127.20.178.67:44161): tablet 794e93a3a8464779a038eab8db5e6b68 (table test-table [id=ec8540527d2d479dbd96a2db9df9d8bd]) successfully deleted
I20260929 03:21:25.463930 7314 catalog_manager.cc:4841] TS 9aa7be32cbc54727b68b1d9c3ce1ab99 (127.20.178.66:35285): tablet 794e93a3a8464779a038eab8db5e6b68 (table test-table [id=ec8540527d2d479dbd96a2db9df9d8bd]) successfully deleted
W20260929 03:21:25.466148 7315 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.20.178.65:39185 (error 108)
I20260929 03:21:25.466303 21193 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuoaHbX/build/release/bin/kudu with pid 7520
I20260929 03:21:25.467290 7789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:25.467148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.65:39185 (local address 127.0.0.1:42962)
0929 03:21:25.467217 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0929 03:21:25.467264 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.178.65:39185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260929 03:21:25.467403 7315 catalog_manager.cc:4584] TS 199fe8c8fad84ac298b6c12756afd961 (127.20.178.65:39185): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 794e93a3a8464779a038eab8db5e6b68: Network error: Client connection negotiation failed: client connection to 127.20.178.65:39185: connect: Connection refused (error 111)
I20260929 03:21:25.470997 21193 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuoaHbX/build/release/bin/kudu with pid 7651
I20260929 03:21:25.475287 21193 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuoaHbX/build/release/bin/kudu with pid 7298
2026-09-29T03:21:25Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (684 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651852476208-21193-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:21:25 dist-test-slave-c9qh krb5kdc[7822](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:21:25 dist-test-slave-c9qh krb5kdc[7822](info): set up 2 sockets
Sep 29 03:21:25 dist-test-slave-c9qh krb5kdc[7822](info): commencing operation
krb5kdc: starting...
W20260929 03:21:26.508460 21193 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.001s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651852476208-21193-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651852476208-21193-0/krb5kdc/test-user.keytab.
Sep 29 03:21:26 dist-test-slave-c9qh krb5kdc[7822](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652086, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:21:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:21:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.178.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.178.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.178.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.126.keytab.
Entry for principal kudu/127.20.178.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.126.keytab.
I20260929 03:21:26.622853 21193 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuoaHbX/build/release/bin/kudu
/tmp/dist-test-taskuoaHbX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651852476208-21193-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651852476208-21193-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651852476208-21193-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651852476208-21193-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.178.126:35341
--webserver_interface=127.20.178.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.126.keytab
--principal=kudu/127.20.178.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.178.84:38771
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.178.126:35341
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651852476208-21193-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651852476208-21193-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651852476208-21193-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:21:26.703222 7835 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:21:26.703414 7835 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:21:26.703437 7835 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:21:26.704589 7835 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:21:26.704664 7835 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:21:26.704726 7835 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:21:26.704752 7835 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:21:26.704787 7835 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:21:26.706252 7835 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.178.84:38771
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651852476208-21193-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651852476208-21193-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.178.126:35341
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.126.keytab
--principal=kudu/127.20.178.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.178.126:35341
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651852476208-21193-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.178.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7835
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651852476208-21193-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:21:26.706511 7835 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:21:26.706748 7835 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:21:26.711434 7843 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:21:26.711442 7840 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:21:26.711664 7835 server_base.cc:1034] running on GCE node
W20260929 03:21:26.711478 7841 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:21:26.711980 7835 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:21:26.712250 7835 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:21:26.713392 7835 hybrid_clock.cc:648] HybridClock initialized: now 1790652086713373 us; error 45 us; skew 500 ppm
Sep 29 03:21:26 dist-test-slave-c9qh krb5kdc[7822](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652086, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:21:26.715736 7835 init.cc:377] Logged in from keytab as kudu/127.20.178.126@KRBTEST.COM (short username kudu)
I20260929 03:21:26.716130 7835 webserver.cc:460] Webserver started at http://127.20.178.126:39727/ using document root <none> and password file <none>
I20260929 03:21:26.716342 7835 fs_manager.cc:359] Metadata directory not provided
I20260929 03:21:26.716390 7835 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:21:26.716562 7835 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:21:26.717592 7835 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651852476208-21193-0/minicluster-data/master-0/data/instance:
uuid: "5ee2ea0c4da841fba478da0624532611"
format_stamp: "Formatted at 2026-09-29 03:21:26 on dist-test-slave-c9qh"
server_key: "ef521249f51db5fafed47cf910815503"
server_key_iv: "15f93439bceefa1aad9d13c83980687c"
server_key_version: "encryptionkey@0"
I20260929 03:21:26.717943 7835 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651852476208-21193-0/minicluster-data/master-0/wal/instance:
uuid: "5ee2ea0c4da841fba478da0624532611"
format_stamp: "Formatted at 2026-09-29 03:21:26 on dist-test-slave-c9qh"
server_key: "ef521249f51db5fafed47cf910815503"
server_key_iv: "15f93439bceefa1aad9d13c83980687c"
server_key_version: "encryptionkey@0"
I20260929 03:21:26.719219 7835 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 03:21:26.719918 7850 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:21:26.720084 7835 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 03:21:26.720178 7835 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651852476208-21193-0/minicluster-data/master-0/data,/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651852476208-21193-0/minicluster-data/master-0/wal
uuid: "5ee2ea0c4da841fba478da0624532611"
format_stamp: "Formatted at 2026-09-29 03:21:26 on dist-test-slave-c9qh"
server_key: "ef521249f51db5fafed47cf910815503"
server_key_iv: "15f93439bceefa1aad9d13c83980687c"
server_key_version: "encryptionkey@0"
I20260929 03:21:26.720258 7835 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651852476208-21193-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651852476208-21193-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651852476208-21193-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:21:26.727877 7835 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:21:26.729612 7835 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:21:26.729763 7835 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:21:26.733464 7835 rpc_server.cc:307] RPC server started. Bound to: 127.20.178.126:35341
I20260929 03:21:26.733495 7902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.178.126:35341 every 8 connection(s)
I20260929 03:21:26.733825 7835 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651852476208-21193-0/minicluster-data/master-0/data/info.pb
I20260929 03:21:26.734650 7903 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:21:26.736789 7903 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ee2ea0c4da841fba478da0624532611: Bootstrap starting.
I20260929 03:21:26.737250 21193 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuoaHbX/build/release/bin/kudu as pid 7835
I20260929 03:21:26.737298 7903 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5ee2ea0c4da841fba478da0624532611: Neither blocks nor log segments found. Creating new log.
I20260929 03:21:26.737367 21193 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651852476208-21193-0/minicluster-data/master-0/wal/instance
I20260929 03:21:26.737562 21193 external_mini_cluster.cc:1428] Setting key c5783863df379fd0d4fe56d33aab7f29
I20260929 03:21:26.737584 7903 log.cc:824] T 00000000000000000000000000000000 P 5ee2ea0c4da841fba478da0624532611: Log is configured to *not* fsync() on all Append() calls
I20260929 03:21:26.738350 7903 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ee2ea0c4da841fba478da0624532611: No bootstrap required, opened a new log
I20260929 03:21:26.740909 7903 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5ee2ea0c4da841fba478da0624532611 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ee2ea0c4da841fba478da0624532611" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 35341 } }
I20260929 03:21:26.741063 7903 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5ee2ea0c4da841fba478da0624532611 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:21:26.741101 7903 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5ee2ea0c4da841fba478da0624532611 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ee2ea0c4da841fba478da0624532611, State: Initialized, Role: FOLLOWER
I20260929 03:21:26.741241 7903 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5ee2ea0c4da841fba478da0624532611 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ee2ea0c4da841fba478da0624532611" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 35341 } }
I20260929 03:21:26.741303 7903 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5ee2ea0c4da841fba478da0624532611 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:21:26.741324 7903 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5ee2ea0c4da841fba478da0624532611 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:21:26.741400 7903 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5ee2ea0c4da841fba478da0624532611 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:21:26.741935 7903 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5ee2ea0c4da841fba478da0624532611 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ee2ea0c4da841fba478da0624532611" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 35341 } }
I20260929 03:21:26.742035 7903 leader_election.cc:302] T 00000000000000000000000000000000 P 5ee2ea0c4da841fba478da0624532611 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5ee2ea0c4da841fba478da0624532611; no voters:
I20260929 03:21:26.742184 7903 leader_election.cc:288] T 00000000000000000000000000000000 P 5ee2ea0c4da841fba478da0624532611 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:21:26.742242 7908 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5ee2ea0c4da841fba478da0624532611 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:21:26.742395 7908 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5ee2ea0c4da841fba478da0624532611 [term 1 LEADER]: Becoming Leader. State: Replica: 5ee2ea0c4da841fba478da0624532611, State: Running, Role: LEADER
I20260929 03:21:26.742414 7903 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5ee2ea0c4da841fba478da0624532611 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:21:26.742480 7908 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5ee2ea0c4da841fba478da0624532611 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ee2ea0c4da841fba478da0624532611" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 35341 } }
I20260929 03:21:26.742796 7909 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ee2ea0c4da841fba478da0624532611 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5ee2ea0c4da841fba478da0624532611" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ee2ea0c4da841fba478da0624532611" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 35341 } } }
I20260929 03:21:26.742887 7909 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ee2ea0c4da841fba478da0624532611 [sys.catalog]: This master's current role is: LEADER
I20260929 03:21:26.743394 7910 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ee2ea0c4da841fba478da0624532611 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5ee2ea0c4da841fba478da0624532611. Latest consensus state: current_term: 1 leader_uuid: "5ee2ea0c4da841fba478da0624532611" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ee2ea0c4da841fba478da0624532611" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 35341 } } }
I20260929 03:21:26.743474 7910 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ee2ea0c4da841fba478da0624532611 [sys.catalog]: This master's current role is: LEADER
I20260929 03:21:26.743628 7918 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:21:26.744163 7918 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 29 03:21:26 dist-test-slave-c9qh krb5kdc[7822](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652086, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.178.126@KRBTEST.COM
I20260929 03:21:26.746745 7918 catalog_manager.cc:1347] Generated new cluster ID: b18496062c5542369c4b21bd439a766e
I20260929 03:21:26.746855 7918 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:21:26.749022 7907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:26.738066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55282 (local address 127.20.178.126:35341)
0929 03:21:26.738262 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0929 03:21:26.738271 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 03:21:26.738302 (+ 31us) server_negotiation.cc:407] Connection header received
0929 03:21:26.738822 (+ 520us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:21:26.738830 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:21:26.738899 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:21:26.739045 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:21:26.739611 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:26.741065 (+ 1454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:26.741955 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:26.742564 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:26.743021 (+ 457us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:21:26.745539 (+ 2518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:21:26.745543 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:21:26.745543 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:21:26.745555 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:21:26.747434 (+ 1879us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:21:26.747869 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:21:26.747871 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:21:26.747872 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:21:26.747952 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:21:26.748168 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:21:26.748169 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:21:26.748169 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:21:26.748390 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:21:26.748547 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0929 03:21:26.748674 (+ 127us) server_negotiation.cc:299] Negotiation successful
0929 03:21:26.748812 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":95,"threads_started":1}
I20260929 03:21:26.755823 7918 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:21:26.756280 7918 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:21:26.763813 7918 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5ee2ea0c4da841fba478da0624532611: Generated new TSK 0
I20260929 03:21:26.764005 7918 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.178.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.178.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.178.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.65.keytab.
Entry for principal kudu/127.20.178.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.65.keytab.
I20260929 03:21:26.789162 21193 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuoaHbX/build/release/bin/kudu
/tmp/dist-test-taskuoaHbX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651852476208-21193-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651852476208-21193-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651852476208-21193-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651852476208-21193-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.178.65:0
--local_ip_for_outbound_sockets=127.20.178.65
--webserver_interface=127.20.178.65
--webserver_port=0
--tserver_master_addrs=127.20.178.126:35341
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.65.keytab
--principal=kudu/127.20.178.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.178.84:38771
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651852476208-21193-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651852476208-21193-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651852476208-21193-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:21:26.869935 7929 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:21:26.870173 7929 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:21:26.870211 7929 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:21:26.871389 7929 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:21:26.871461 7929 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:21:26.871506 7929 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.178.65
I20260929 03:21:26.872996 7929 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.178.84:38771
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651852476208-21193-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651852476208-21193-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.65.keytab
--principal=kudu/127.20.178.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.178.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651852476208-21193-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.178.65
--webserver_port=0
--tserver_master_addrs=127.20.178.126:35341
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7929
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.178.65
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651852476208-21193-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:21:26.873303 7929 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:21:26.873540 7929 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:21:26.878568 7934 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:21:26.878564 7935 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:21:26.878605 7937 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:21:26.878856 7929 server_base.cc:1034] running on GCE node
I20260929 03:21:26.879046 7929 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:21:26.879241 7929 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:21:26.880445 7929 hybrid_clock.cc:648] HybridClock initialized: now 1790652086880432 us; error 37 us; skew 500 ppm
Sep 29 03:21:26 dist-test-slave-c9qh krb5kdc[7822](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652086, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:21:26.882937 7929 init.cc:377] Logged in from keytab as kudu/127.20.178.65@KRBTEST.COM (short username kudu)
I20260929 03:21:26.883394 7929 webserver.cc:460] Webserver started at http://127.20.178.65:41231/ using document root <none> and password file <none>
I20260929 03:21:26.883639 7929 fs_manager.cc:359] Metadata directory not provided
I20260929 03:21:26.883706 7929 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:21:26.883852 7929 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:21:26.884936 7929 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651852476208-21193-0/minicluster-data/ts-0/data/instance:
uuid: "f276fcb82f3041219689f781013028f4"
format_stamp: "Formatted at 2026-09-29 03:21:26 on dist-test-slave-c9qh"
server_key: "92015a1da2f2a6ccbe0154595e455bff"
server_key_iv: "d84278008c46dcf5d24231834107f418"
server_key_version: "encryptionkey@0"
I20260929 03:21:26.885284 7929 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651852476208-21193-0/minicluster-data/ts-0/wal/instance:
uuid: "f276fcb82f3041219689f781013028f4"
format_stamp: "Formatted at 2026-09-29 03:21:26 on dist-test-slave-c9qh"
server_key: "92015a1da2f2a6ccbe0154595e455bff"
server_key_iv: "d84278008c46dcf5d24231834107f418"
server_key_version: "encryptionkey@0"
I20260929 03:21:26.886569 7929 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 03:21:26.887272 7944 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:21:26.887444 7929 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 03:21:26.887543 7929 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651852476208-21193-0/minicluster-data/ts-0/data,/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651852476208-21193-0/minicluster-data/ts-0/wal
uuid: "f276fcb82f3041219689f781013028f4"
format_stamp: "Formatted at 2026-09-29 03:21:26 on dist-test-slave-c9qh"
server_key: "92015a1da2f2a6ccbe0154595e455bff"
server_key_iv: "d84278008c46dcf5d24231834107f418"
server_key_version: "encryptionkey@0"
I20260929 03:21:26.887622 7929 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651852476208-21193-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651852476208-21193-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651852476208-21193-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:21:26.894702 7929 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:21:26.896471 7929 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:21:26.896705 7929 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:21:26.896935 7929 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:21:26.897246 7929 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:21:26.897300 7929 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:21:26.897343 7929 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:21:26.897369 7929 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:21:26.903633 7929 rpc_server.cc:307] RPC server started. Bound to: 127.20.178.65:41855
I20260929 03:21:26.903680 8057 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.178.65:41855 every 8 connection(s)
I20260929 03:21:26.904037 7929 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651852476208-21193-0/minicluster-data/ts-0/data/info.pb
Sep 29 03:21:26 dist-test-slave-c9qh krb5kdc[7822](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652086, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.65@KRBTEST.COM for kudu/127.20.178.126@KRBTEST.COM
I20260929 03:21:26.912787 7907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:26.904850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.65:60327 (local address 127.20.178.126:35341)
0929 03:21:26.904930 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0929 03:21:26.904933 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:21:26.905129 (+ 196us) server_negotiation.cc:407] Connection header received
0929 03:21:26.905624 (+ 495us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:21:26.905625 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:21:26.905634 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:21:26.905687 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:21:26.906590 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:26.907085 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:26.908106 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:26.908308 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:26.908366 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:21:26.910417 (+ 2051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:21:26.910418 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:21:26.910418 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:21:26.910425 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:21:26.911584 (+ 1159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:21:26.911955 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:21:26.911956 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:21:26.911956 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:21:26.912007 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:21:26.912218 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:21:26.912218 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:21:26.912219 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:21:26.912317 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:21:26.912367 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0929 03:21:26.912595 (+ 228us) server_negotiation.cc:299] Negotiation successful
0929 03:21:26.912721 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 03:21:26.912950 8060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:26.904824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:35341 (local address 127.20.178.65:60327)
0929 03:21:26.905006 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0929 03:21:26.905042 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0929 03:21:26.905535 (+ 493us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:21:26.905751 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:21:26.905754 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:21:26.906317 (+ 563us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:21:26.906514 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:21:26.906518 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:21:26.907206 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:21:26.907207 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:21:26.908028 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:21:26.908030 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:21:26.908386 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:21:26.908387 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:21:26.908485 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:21:26.908968 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:21:26.908976 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:21:26.910325 (+ 1349us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:21:26.911657 (+ 1332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:21:26.911663 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:21:26.911664 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:21:26.911884 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:21:26.912069 (+ 185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:21:26.912070 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:21:26.912070 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:21:26.912165 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:21:26.912374 (+ 209us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:21:26.912378 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:21:26.912489 (+ 111us) client_negotiation.cc:769] Sending connection context
0929 03:21:26.912570 (+ 81us) client_negotiation.cc:240] Negotiation successful
0929 03:21:26.912744 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":98,"threads_started":1}
I20260929 03:21:26.913477 8058 heartbeater.cc:347] Connected to a master server at 127.20.178.126:35341
I20260929 03:21:26.913568 8058 heartbeater.cc:464] Registering TS with master...
I20260929 03:21:26.913704 8058 heartbeater.cc:511] Master 127.20.178.126:35341 requested a full tablet report, sending...
I20260929 03:21:26.914175 7867 ts_manager.cc:194] Registered new tserver with Master: f276fcb82f3041219689f781013028f4 (127.20.178.65:41855)
I20260929 03:21:26.914199 21193 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuoaHbX/build/release/bin/kudu as pid 7929
I20260929 03:21:26.914307 21193 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651852476208-21193-0/minicluster-data/ts-0/wal/instance
I20260929 03:21:26.914510 21193 external_mini_cluster.cc:1428] Setting key b82b703788d88ce6942b7e73746f71d5
I20260929 03:21:26.915025 7867 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.178.65@KRBTEST.COM'} at 127.20.178.65:60327
WARNING: no policy specified for kudu/127.20.178.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.178.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.178.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.66.keytab.
Entry for principal kudu/127.20.178.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.66.keytab.
I20260929 03:21:26.939312 21193 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuoaHbX/build/release/bin/kudu
/tmp/dist-test-taskuoaHbX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651852476208-21193-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651852476208-21193-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651852476208-21193-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651852476208-21193-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.178.66:0
--local_ip_for_outbound_sockets=127.20.178.66
--webserver_interface=127.20.178.66
--webserver_port=0
--tserver_master_addrs=127.20.178.126:35341
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.66.keytab
--principal=kudu/127.20.178.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.178.84:38771
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651852476208-21193-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651852476208-21193-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651852476208-21193-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:21:27.019637 8063 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:21:27.019834 8063 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:21:27.019856 8063 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:21:27.021009 8063 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:21:27.021051 8063 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:21:27.021077 8063 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.178.66
I20260929 03:21:27.022523 8063 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.178.84:38771
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651852476208-21193-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651852476208-21193-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.66.keytab
--principal=kudu/127.20.178.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.178.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651852476208-21193-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.178.66
--webserver_port=0
--tserver_master_addrs=127.20.178.126:35341
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8063
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.178.66
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651852476208-21193-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:21:27.022760 8063 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:21:27.023016 8063 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:21:27.027865 8068 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:21:27.027915 8071 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:21:27.027877 8069 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:21:27.028074 8063 server_base.cc:1034] running on GCE node
I20260929 03:21:27.028256 8063 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:21:27.028462 8063 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:21:27.029593 8063 hybrid_clock.cc:648] HybridClock initialized: now 1790652087029582 us; error 26 us; skew 500 ppm
Sep 29 03:21:27 dist-test-slave-c9qh krb5kdc[7822](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652087, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:21:27.031930 8063 init.cc:377] Logged in from keytab as kudu/127.20.178.66@KRBTEST.COM (short username kudu)
I20260929 03:21:27.032349 8063 webserver.cc:460] Webserver started at http://127.20.178.66:34769/ using document root <none> and password file <none>
I20260929 03:21:27.032589 8063 fs_manager.cc:359] Metadata directory not provided
I20260929 03:21:27.032680 8063 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:21:27.032830 8063 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:21:27.033864 8063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651852476208-21193-0/minicluster-data/ts-1/data/instance:
uuid: "e86cc27863094a27a902cee18e7567c2"
format_stamp: "Formatted at 2026-09-29 03:21:27 on dist-test-slave-c9qh"
server_key: "4bebbc46b47908a61b802a553af74ca8"
server_key_iv: "3fa191617b163cd5f2cf4d8a4a285792"
server_key_version: "encryptionkey@0"
I20260929 03:21:27.034257 8063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651852476208-21193-0/minicluster-data/ts-1/wal/instance:
uuid: "e86cc27863094a27a902cee18e7567c2"
format_stamp: "Formatted at 2026-09-29 03:21:27 on dist-test-slave-c9qh"
server_key: "4bebbc46b47908a61b802a553af74ca8"
server_key_iv: "3fa191617b163cd5f2cf4d8a4a285792"
server_key_version: "encryptionkey@0"
I20260929 03:21:27.035588 8063 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 03:21:27.036291 8078 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:21:27.036471 8063 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 03:21:27.036566 8063 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651852476208-21193-0/minicluster-data/ts-1/data,/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651852476208-21193-0/minicluster-data/ts-1/wal
uuid: "e86cc27863094a27a902cee18e7567c2"
format_stamp: "Formatted at 2026-09-29 03:21:27 on dist-test-slave-c9qh"
server_key: "4bebbc46b47908a61b802a553af74ca8"
server_key_iv: "3fa191617b163cd5f2cf4d8a4a285792"
server_key_version: "encryptionkey@0"
I20260929 03:21:27.036669 8063 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651852476208-21193-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651852476208-21193-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651852476208-21193-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:21:27.059264 8063 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:21:27.061357 8063 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:21:27.061540 8063 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:21:27.061781 8063 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:21:27.062109 8063 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:21:27.062165 8063 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:21:27.062206 8063 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:21:27.062234 8063 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:21:27.068007 8063 rpc_server.cc:307] RPC server started. Bound to: 127.20.178.66:34733
I20260929 03:21:27.068056 8191 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.178.66:34733 every 8 connection(s)
I20260929 03:21:27.068377 8063 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651852476208-21193-0/minicluster-data/ts-1/data/info.pb
I20260929 03:21:27.073817 21193 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuoaHbX/build/release/bin/kudu as pid 8063
I20260929 03:21:27.073925 21193 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651852476208-21193-0/minicluster-data/ts-1/wal/instance
I20260929 03:21:27.074107 21193 external_mini_cluster.cc:1428] Setting key 61c1966c9e53228c31aa007f10dd6682
Sep 29 03:21:27 dist-test-slave-c9qh krb5kdc[7822](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652087, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.66@KRBTEST.COM for kudu/127.20.178.126@KRBTEST.COM
I20260929 03:21:27.078095 7907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:27.069157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.66:44597 (local address 127.20.178.126:35341)
0929 03:21:27.069203 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0929 03:21:27.069205 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:21:27.069417 (+ 212us) server_negotiation.cc:407] Connection header received
0929 03:21:27.069879 (+ 462us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:21:27.069880 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:21:27.069890 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:21:27.069948 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:21:27.070896 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:27.071387 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:27.072350 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:27.072530 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:27.072581 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:21:27.075047 (+ 2466us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:21:27.075049 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:21:27.075049 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:21:27.075057 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:21:27.076580 (+ 1523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:21:27.077127 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:21:27.077128 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:21:27.077128 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:21:27.077180 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:21:27.077470 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:21:27.077471 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:21:27.077471 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:21:27.077576 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:21:27.077628 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0929 03:21:27.077957 (+ 329us) server_negotiation.cc:299] Negotiation successful
0929 03:21:27.078031 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260929 03:21:27.078418 8194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:27.069134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:35341 (local address 127.20.178.66:44597)
0929 03:21:27.069292 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0929 03:21:27.069328 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0929 03:21:27.069797 (+ 469us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:21:27.070004 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:21:27.070007 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:21:27.070613 (+ 606us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:21:27.070818 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:21:27.070823 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:21:27.071467 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:21:27.071467 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:21:27.072265 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:21:27.072268 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:21:27.072601 (+ 333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:21:27.072602 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:21:27.072775 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:21:27.073237 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:21:27.073247 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:21:27.074951 (+ 1704us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:21:27.076712 (+ 1761us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:21:27.076719 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:21:27.076722 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:21:27.077050 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:21:27.077248 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:21:27.077249 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:21:27.077250 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:21:27.077403 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:21:27.077644 (+ 241us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:21:27.077650 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:21:27.077819 (+ 169us) client_negotiation.cc:769] Sending connection context
0929 03:21:27.077936 (+ 117us) client_negotiation.cc:240] Negotiation successful
0929 03:21:27.078121 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":82,"threads_started":1}
I20260929 03:21:27.078966 8192 heartbeater.cc:347] Connected to a master server at 127.20.178.126:35341
I20260929 03:21:27.079077 8192 heartbeater.cc:464] Registering TS with master...
I20260929 03:21:27.079275 8192 heartbeater.cc:511] Master 127.20.178.126:35341 requested a full tablet report, sending...
I20260929 03:21:27.079720 7867 ts_manager.cc:194] Registered new tserver with Master: e86cc27863094a27a902cee18e7567c2 (127.20.178.66:34733)
I20260929 03:21:27.080106 7867 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.178.66@KRBTEST.COM'} at 127.20.178.66:44597
WARNING: no policy specified for kudu/127.20.178.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.178.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.178.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.67.keytab.
Entry for principal kudu/127.20.178.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.67.keytab.
I20260929 03:21:27.099522 21193 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuoaHbX/build/release/bin/kudu
/tmp/dist-test-taskuoaHbX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651852476208-21193-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651852476208-21193-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651852476208-21193-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651852476208-21193-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.178.67:0
--local_ip_for_outbound_sockets=127.20.178.67
--webserver_interface=127.20.178.67
--webserver_port=0
--tserver_master_addrs=127.20.178.126:35341
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.67.keytab
--principal=kudu/127.20.178.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.178.84:38771
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651852476208-21193-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651852476208-21193-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651852476208-21193-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:21:27.178630 8197 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:21:27.178876 8197 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:21:27.178917 8197 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:21:27.180058 8197 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:21:27.180128 8197 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:21:27.180174 8197 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.178.67
I20260929 03:21:27.181702 8197 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.178.84:38771
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651852476208-21193-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651852476208-21193-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.67.keytab
--principal=kudu/127.20.178.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.178.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651852476208-21193-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.178.67
--webserver_port=0
--tserver_master_addrs=127.20.178.126:35341
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8197
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.178.67
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651852476208-21193-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:21:27.181982 8197 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:21:27.182207 8197 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:21:27.187134 8203 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:21:27.187155 8205 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:21:27.187197 8197 server_base.cc:1034] running on GCE node
W20260929 03:21:27.187166 8202 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:21:27.187522 8197 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:21:27.187747 8197 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:21:27.188905 8197 hybrid_clock.cc:648] HybridClock initialized: now 1790652087188892 us; error 25 us; skew 500 ppm
Sep 29 03:21:27 dist-test-slave-c9qh krb5kdc[7822](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652087, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:21:27.191466 8197 init.cc:377] Logged in from keytab as kudu/127.20.178.67@KRBTEST.COM (short username kudu)
I20260929 03:21:27.191864 8197 webserver.cc:460] Webserver started at http://127.20.178.67:45305/ using document root <none> and password file <none>
I20260929 03:21:27.192059 8197 fs_manager.cc:359] Metadata directory not provided
I20260929 03:21:27.192099 8197 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:21:27.192183 8197 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:21:27.193318 8197 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651852476208-21193-0/minicluster-data/ts-2/data/instance:
uuid: "89b68eb29a8644d38d3ca00ad67d532a"
format_stamp: "Formatted at 2026-09-29 03:21:27 on dist-test-slave-c9qh"
server_key: "9ae5c56a4f02e3f520a18977122c3cab"
server_key_iv: "9fd26264350f697d6ad36cc2a661cbca"
server_key_version: "encryptionkey@0"
I20260929 03:21:27.193655 8197 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651852476208-21193-0/minicluster-data/ts-2/wal/instance:
uuid: "89b68eb29a8644d38d3ca00ad67d532a"
format_stamp: "Formatted at 2026-09-29 03:21:27 on dist-test-slave-c9qh"
server_key: "9ae5c56a4f02e3f520a18977122c3cab"
server_key_iv: "9fd26264350f697d6ad36cc2a661cbca"
server_key_version: "encryptionkey@0"
I20260929 03:21:27.194950 8197 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 03:21:27.195611 8212 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:21:27.195811 8197 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 03:21:27.195901 8197 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651852476208-21193-0/minicluster-data/ts-2/data,/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651852476208-21193-0/minicluster-data/ts-2/wal
uuid: "89b68eb29a8644d38d3ca00ad67d532a"
format_stamp: "Formatted at 2026-09-29 03:21:27 on dist-test-slave-c9qh"
server_key: "9ae5c56a4f02e3f520a18977122c3cab"
server_key_iv: "9fd26264350f697d6ad36cc2a661cbca"
server_key_version: "encryptionkey@0"
I20260929 03:21:27.195952 8197 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651852476208-21193-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651852476208-21193-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651852476208-21193-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:21:27.207247 8197 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:21:27.209445 8197 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:21:27.209549 8197 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:21:27.209704 8197 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:21:27.209951 8197 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:21:27.209981 8197 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:21:27.210001 8197 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:21:27.210014 8197 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:21:27.215457 8197 rpc_server.cc:307] RPC server started. Bound to: 127.20.178.67:42631
I20260929 03:21:27.215528 8325 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.178.67:42631 every 8 connection(s)
I20260929 03:21:27.215835 8197 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651852476208-21193-0/minicluster-data/ts-2/data/info.pb
Sep 29 03:21:27 dist-test-slave-c9qh krb5kdc[7822](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652087, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.67@KRBTEST.COM for kudu/127.20.178.126@KRBTEST.COM
I20260929 03:21:27.224156 21193 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuoaHbX/build/release/bin/kudu as pid 8197
I20260929 03:21:27.224264 21193 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651852476208-21193-0/minicluster-data/ts-2/wal/instance
I20260929 03:21:27.224481 21193 external_mini_cluster.cc:1428] Setting key b0cfef406528c9df0a8ba35d38061681
I20260929 03:21:27.224743 7907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:27.216544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.67:57019 (local address 127.20.178.126:35341)
0929 03:21:27.216600 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0929 03:21:27.216603 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:21:27.216859 (+ 256us) server_negotiation.cc:407] Connection header received
0929 03:21:27.217290 (+ 431us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:21:27.217291 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:21:27.217301 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:21:27.217352 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:21:27.218212 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:27.218700 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:27.219621 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:27.219797 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:27.219845 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:21:27.221931 (+ 2086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:21:27.221933 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:21:27.221933 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:21:27.221939 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:21:27.223297 (+ 1358us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:21:27.223644 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:21:27.223645 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:21:27.223645 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:21:27.223723 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:21:27.223992 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:21:27.223993 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:21:27.223993 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:21:27.224103 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:21:27.224174 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0929 03:21:27.224510 (+ 336us) server_negotiation.cc:299] Negotiation successful
0929 03:21:27.224584 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260929 03:21:27.224902 8328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:27.216516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:35341 (local address 127.20.178.67:57019)
0929 03:21:27.216727 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0929 03:21:27.216768 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 03:21:27.217211 (+ 443us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:21:27.217406 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:21:27.217409 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:21:27.217950 (+ 541us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:21:27.218128 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:21:27.218132 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:21:27.218781 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:21:27.218782 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:21:27.219551 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:21:27.219553 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:21:27.219879 (+ 326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:21:27.219879 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:21:27.220026 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:21:27.220458 (+ 432us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:21:27.220466 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:21:27.221848 (+ 1382us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:21:27.223370 (+ 1522us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:21:27.223375 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:21:27.223377 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:21:27.223581 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:21:27.223792 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:21:27.223793 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:21:27.223794 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:21:27.223913 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:21:27.224177 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:21:27.224184 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:21:27.224359 (+ 175us) client_negotiation.cc:769] Sending connection context
0929 03:21:27.224488 (+ 129us) client_negotiation.cc:240] Negotiation successful
0929 03:21:27.224669 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":112,"threads_started":1}
I20260929 03:21:27.225358 8326 heartbeater.cc:347] Connected to a master server at 127.20.178.126:35341
I20260929 03:21:27.225476 8326 heartbeater.cc:464] Registering TS with master...
I20260929 03:21:27.225639 8326 heartbeater.cc:511] Master 127.20.178.126:35341 requested a full tablet report, sending...
I20260929 03:21:27.225996 7866 ts_manager.cc:194] Registered new tserver with Master: 89b68eb29a8644d38d3ca00ad67d532a (127.20.178.67:42631)
I20260929 03:21:27.226323 7866 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.178.67@KRBTEST.COM'} at 127.20.178.67:57019
I20260929 03:21:27.236316 21193 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 03:21:27.244311 7907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:27.237782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55296 (local address 127.20.178.126:35341)
0929 03:21:27.237858 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0929 03:21:27.237861 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:21:27.237958 (+ 97us) server_negotiation.cc:407] Connection header received
0929 03:21:27.238027 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:21:27.238028 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:21:27.238038 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:21:27.238093 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:21:27.238680 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:27.239208 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:27.239972 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:27.240174 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:27.240261 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:21:27.241579 (+ 1318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:21:27.241581 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:21:27.241581 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:21:27.241588 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:21:27.243008 (+ 1420us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:21:27.243442 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:21:27.243443 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:21:27.243443 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:21:27.243495 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:21:27.243715 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:21:27.243716 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:21:27.243717 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:21:27.243875 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:21:27.244019 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0929 03:21:27.244105 (+ 86us) server_negotiation.cc:299] Negotiation successful
0929 03:21:27.244202 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 03:21:27.245743 7866 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55296:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 03:21:27.247220 7866 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 03:21:27.253544 8336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:27.249616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.67:42631 (local address 127.0.0.1:55070)
0929 03:21:27.249768 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0929 03:21:27.249777 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 03:21:27.249824 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:21:27.250403 (+ 579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:21:27.250404 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:21:27.250406 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:21:27.250443 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:21:27.250444 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:21:27.251598 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:21:27.251599 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:21:27.252507 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:21:27.252510 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:21:27.253229 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:21:27.253239 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:21:27.253316 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:21:27.253330 (+ 14us) client_negotiation.cc:769] Sending connection context
0929 03:21:27.253405 (+ 75us) client_negotiation.cc:240] Negotiation successful
0929 03:21:27.253460 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"mutex_wait_us":2,"thread_start_us":53,"threads_started":1}
I20260929 03:21:27.253734 8334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:27.249467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.65:41855 (local address 127.0.0.1:47246)
0929 03:21:27.249577 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0929 03:21:27.249619 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 03:21:27.249679 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:21:27.250127 (+ 448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:21:27.250130 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:21:27.250133 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:21:27.250203 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:21:27.250206 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:21:27.251197 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:21:27.251199 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:21:27.252443 (+ 1244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:21:27.252445 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:21:27.253378 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:21:27.253379 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:21:27.253434 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:21:27.253437 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 03:21:27.253496 (+ 59us) client_negotiation.cc:240] Negotiation successful
0929 03:21:27.253528 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":47,"threads_started":1}
I20260929 03:21:27.253822 8339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:27.250051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55070 (local address 127.20.178.67:42631)
0929 03:21:27.250233 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0929 03:21:27.250238 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:21:27.250246 (+ 8us) server_negotiation.cc:407] Connection header received
0929 03:21:27.250283 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:21:27.250285 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:21:27.250301 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:21:27.250392 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:21:27.250506 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:27.251521 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:27.252589 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:27.253130 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:27.253218 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:21:27.253282 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0929 03:21:27.253668 (+ 386us) server_negotiation.cc:299] Negotiation successful
0929 03:21:27.253721 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":75,"threads_started":1}
I20260929 03:21:27.253867 8338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:27.249744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47246 (local address 127.20.178.65:41855)
0929 03:21:27.249918 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0929 03:21:27.249923 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:21:27.249934 (+ 11us) server_negotiation.cc:407] Connection header received
0929 03:21:27.249977 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:21:27.249979 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:21:27.250013 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:21:27.250285 (+ 272us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:21:27.250390 (+ 105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:27.251098 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:27.252531 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:27.253285 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:27.253381 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:21:27.253441 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0929 03:21:27.253512 (+ 71us) server_negotiation.cc:299] Negotiation successful
0929 03:21:27.253553 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":108,"threads_started":1}
I20260929 03:21:27.254091 8337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:27.249595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34112 (local address 127.20.178.66:34733)
0929 03:21:27.249921 (+ 326us) server_negotiation.cc:206] Beginning negotiation
0929 03:21:27.249925 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:21:27.249934 (+ 9us) server_negotiation.cc:407] Connection header received
0929 03:21:27.249965 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:21:27.249967 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:21:27.250001 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:21:27.250076 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:21:27.250309 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:27.251439 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:27.252678 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:27.253357 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:27.253416 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:21:27.253463 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0929 03:21:27.253995 (+ 532us) server_negotiation.cc:299] Negotiation successful
0929 03:21:27.254033 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":91,"threads_started":1}
I20260929 03:21:27.254299 8335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:27.249407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.66:34733 (local address 127.0.0.1:34112)
0929 03:21:27.249568 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0929 03:21:27.249616 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0929 03:21:27.249680 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:21:27.250166 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:21:27.250167 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:21:27.250169 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:21:27.250227 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:21:27.250228 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:21:27.251544 (+ 1316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:21:27.251545 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:21:27.252587 (+ 1042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:21:27.252589 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:21:27.253500 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:21:27.253501 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:21:27.253561 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:21:27.253565 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 03:21:27.253744 (+ 179us) client_negotiation.cc:240] Negotiation successful
0929 03:21:27.254233 (+ 489us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":68,"threads_started":1}
I20260929 03:21:27.254566 8260 tablet_service.cc:1467] Processing CreateTablet for tablet 1d22cf21b3fb4f5e840598f5002267a3 (DEFAULT_TABLE table=test-table [id=3745aa5c4a084ee4acaad65ee39f443b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:21:27.254582 7992 tablet_service.cc:1467] Processing CreateTablet for tablet 1d22cf21b3fb4f5e840598f5002267a3 (DEFAULT_TABLE table=test-table [id=3745aa5c4a084ee4acaad65ee39f443b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:21:27.254931 8260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d22cf21b3fb4f5e840598f5002267a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:21:27.254942 7992 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d22cf21b3fb4f5e840598f5002267a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:21:27.254987 8126 tablet_service.cc:1467] Processing CreateTablet for tablet 1d22cf21b3fb4f5e840598f5002267a3 (DEFAULT_TABLE table=test-table [id=3745aa5c4a084ee4acaad65ee39f443b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:21:27.255272 8126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d22cf21b3fb4f5e840598f5002267a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:21:27.257530 8340 tablet_bootstrap.cc:492] T 1d22cf21b3fb4f5e840598f5002267a3 P f276fcb82f3041219689f781013028f4: Bootstrap starting.
I20260929 03:21:27.258186 8341 tablet_bootstrap.cc:492] T 1d22cf21b3fb4f5e840598f5002267a3 P 89b68eb29a8644d38d3ca00ad67d532a: Bootstrap starting.
I20260929 03:21:27.258332 8340 tablet_bootstrap.cc:654] T 1d22cf21b3fb4f5e840598f5002267a3 P f276fcb82f3041219689f781013028f4: Neither blocks nor log segments found. Creating new log.
I20260929 03:21:27.258417 8342 tablet_bootstrap.cc:492] T 1d22cf21b3fb4f5e840598f5002267a3 P e86cc27863094a27a902cee18e7567c2: Bootstrap starting.
I20260929 03:21:27.258870 8341 tablet_bootstrap.cc:654] T 1d22cf21b3fb4f5e840598f5002267a3 P 89b68eb29a8644d38d3ca00ad67d532a: Neither blocks nor log segments found. Creating new log.
I20260929 03:21:27.258912 8340 log.cc:824] T 1d22cf21b3fb4f5e840598f5002267a3 P f276fcb82f3041219689f781013028f4: Log is configured to *not* fsync() on all Append() calls
I20260929 03:21:27.258953 8342 tablet_bootstrap.cc:654] T 1d22cf21b3fb4f5e840598f5002267a3 P e86cc27863094a27a902cee18e7567c2: Neither blocks nor log segments found. Creating new log.
I20260929 03:21:27.259351 8341 log.cc:824] T 1d22cf21b3fb4f5e840598f5002267a3 P 89b68eb29a8644d38d3ca00ad67d532a: Log is configured to *not* fsync() on all Append() calls
I20260929 03:21:27.259521 8342 log.cc:824] T 1d22cf21b3fb4f5e840598f5002267a3 P e86cc27863094a27a902cee18e7567c2: Log is configured to *not* fsync() on all Append() calls
I20260929 03:21:27.259754 8340 tablet_bootstrap.cc:492] T 1d22cf21b3fb4f5e840598f5002267a3 P f276fcb82f3041219689f781013028f4: No bootstrap required, opened a new log
I20260929 03:21:27.259827 8340 ts_tablet_manager.cc:1397] T 1d22cf21b3fb4f5e840598f5002267a3 P f276fcb82f3041219689f781013028f4: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 03:21:27.260046 8341 tablet_bootstrap.cc:492] T 1d22cf21b3fb4f5e840598f5002267a3 P 89b68eb29a8644d38d3ca00ad67d532a: No bootstrap required, opened a new log
I20260929 03:21:27.260119 8341 ts_tablet_manager.cc:1397] T 1d22cf21b3fb4f5e840598f5002267a3 P 89b68eb29a8644d38d3ca00ad67d532a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 03:21:27.260356 8342 tablet_bootstrap.cc:492] T 1d22cf21b3fb4f5e840598f5002267a3 P e86cc27863094a27a902cee18e7567c2: No bootstrap required, opened a new log
I20260929 03:21:27.260471 8342 ts_tablet_manager.cc:1397] T 1d22cf21b3fb4f5e840598f5002267a3 P e86cc27863094a27a902cee18e7567c2: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 03:21:27.261409 8340 raft_consensus.cc:348] T 1d22cf21b3fb4f5e840598f5002267a3 P f276fcb82f3041219689f781013028f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e86cc27863094a27a902cee18e7567c2" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 34733 } } peers { permanent_uuid: "89b68eb29a8644d38d3ca00ad67d532a" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 42631 } } peers { permanent_uuid: "f276fcb82f3041219689f781013028f4" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 41855 } }
I20260929 03:21:27.261579 8340 raft_consensus.cc:374] T 1d22cf21b3fb4f5e840598f5002267a3 P f276fcb82f3041219689f781013028f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:21:27.261623 8340 raft_consensus.cc:729] T 1d22cf21b3fb4f5e840598f5002267a3 P f276fcb82f3041219689f781013028f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f276fcb82f3041219689f781013028f4, State: Initialized, Role: FOLLOWER
I20260929 03:21:27.261736 8340 consensus_queue.cc:260] T 1d22cf21b3fb4f5e840598f5002267a3 P f276fcb82f3041219689f781013028f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e86cc27863094a27a902cee18e7567c2" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 34733 } } peers { permanent_uuid: "89b68eb29a8644d38d3ca00ad67d532a" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 42631 } } peers { permanent_uuid: "f276fcb82f3041219689f781013028f4" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 41855 } }
I20260929 03:21:27.262024 8058 heartbeater.cc:503] Master 127.20.178.126:35341 was elected leader, sending a full tablet report...
I20260929 03:21:27.262060 8341 raft_consensus.cc:348] T 1d22cf21b3fb4f5e840598f5002267a3 P 89b68eb29a8644d38d3ca00ad67d532a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e86cc27863094a27a902cee18e7567c2" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 34733 } } peers { permanent_uuid: "89b68eb29a8644d38d3ca00ad67d532a" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 42631 } } peers { permanent_uuid: "f276fcb82f3041219689f781013028f4" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 41855 } }
I20260929 03:21:27.262234 8341 raft_consensus.cc:374] T 1d22cf21b3fb4f5e840598f5002267a3 P 89b68eb29a8644d38d3ca00ad67d532a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:21:27.262270 8341 raft_consensus.cc:729] T 1d22cf21b3fb4f5e840598f5002267a3 P 89b68eb29a8644d38d3ca00ad67d532a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89b68eb29a8644d38d3ca00ad67d532a, State: Initialized, Role: FOLLOWER
I20260929 03:21:27.262310 8340 ts_tablet_manager.cc:1428] T 1d22cf21b3fb4f5e840598f5002267a3 P f276fcb82f3041219689f781013028f4: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260929 03:21:27.262400 8341 consensus_queue.cc:260] T 1d22cf21b3fb4f5e840598f5002267a3 P 89b68eb29a8644d38d3ca00ad67d532a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e86cc27863094a27a902cee18e7567c2" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 34733 } } peers { permanent_uuid: "89b68eb29a8644d38d3ca00ad67d532a" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 42631 } } peers { permanent_uuid: "f276fcb82f3041219689f781013028f4" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 41855 } }
I20260929 03:21:27.262400 8342 raft_consensus.cc:348] T 1d22cf21b3fb4f5e840598f5002267a3 P e86cc27863094a27a902cee18e7567c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e86cc27863094a27a902cee18e7567c2" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 34733 } } peers { permanent_uuid: "89b68eb29a8644d38d3ca00ad67d532a" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 42631 } } peers { permanent_uuid: "f276fcb82f3041219689f781013028f4" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 41855 } }
I20260929 03:21:27.262554 8342 raft_consensus.cc:374] T 1d22cf21b3fb4f5e840598f5002267a3 P e86cc27863094a27a902cee18e7567c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:21:27.262591 8342 raft_consensus.cc:729] T 1d22cf21b3fb4f5e840598f5002267a3 P e86cc27863094a27a902cee18e7567c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e86cc27863094a27a902cee18e7567c2, State: Initialized, Role: FOLLOWER
I20260929 03:21:27.262671 8341 ts_tablet_manager.cc:1428] T 1d22cf21b3fb4f5e840598f5002267a3 P 89b68eb29a8644d38d3ca00ad67d532a: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260929 03:21:27.262699 8326 heartbeater.cc:503] Master 127.20.178.126:35341 was elected leader, sending a full tablet report...
I20260929 03:21:27.262692 8342 consensus_queue.cc:260] T 1d22cf21b3fb4f5e840598f5002267a3 P e86cc27863094a27a902cee18e7567c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e86cc27863094a27a902cee18e7567c2" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 34733 } } peers { permanent_uuid: "89b68eb29a8644d38d3ca00ad67d532a" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 42631 } } peers { permanent_uuid: "f276fcb82f3041219689f781013028f4" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 41855 } }
I20260929 03:21:27.262961 8342 ts_tablet_manager.cc:1428] T 1d22cf21b3fb4f5e840598f5002267a3 P e86cc27863094a27a902cee18e7567c2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 03:21:27.262974 8192 heartbeater.cc:503] Master 127.20.178.126:35341 was elected leader, sending a full tablet report...
W20260929 03:21:27.318939 8193 tablet.cc:2369] T 1d22cf21b3fb4f5e840598f5002267a3 P e86cc27863094a27a902cee18e7567c2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 03:21:27.364933 8347 raft_consensus.cc:482] T 1d22cf21b3fb4f5e840598f5002267a3 P 89b68eb29a8644d38d3ca00ad67d532a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:21:27.365067 8347 raft_consensus.cc:504] T 1d22cf21b3fb4f5e840598f5002267a3 P 89b68eb29a8644d38d3ca00ad67d532a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e86cc27863094a27a902cee18e7567c2" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 34733 } } peers { permanent_uuid: "89b68eb29a8644d38d3ca00ad67d532a" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 42631 } } peers { permanent_uuid: "f276fcb82f3041219689f781013028f4" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 41855 } }
I20260929 03:21:27.365384 8347 leader_election.cc:288] T 1d22cf21b3fb4f5e840598f5002267a3 P 89b68eb29a8644d38d3ca00ad67d532a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e86cc27863094a27a902cee18e7567c2 (127.20.178.66:34733), f276fcb82f3041219689f781013028f4 (127.20.178.65:41855)
I20260929 03:21:27.368546 8338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:27.365692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.67:44005 (local address 127.20.178.65:41855)
0929 03:21:27.365736 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0929 03:21:27.365738 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:21:27.365779 (+ 41us) server_negotiation.cc:407] Connection header received
0929 03:21:27.365802 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:21:27.365803 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:21:27.365812 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:21:27.365886 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:21:27.366077 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:27.366653 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:27.367637 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:27.368187 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:27.368243 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:21:27.368251 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0929 03:21:27.368426 (+ 175us) server_negotiation.cc:299] Negotiation successful
0929 03:21:27.368469 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260929 03:21:27.368566 8337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:27.365666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.67:55427 (local address 127.20.178.66:34733)
0929 03:21:27.365737 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0929 03:21:27.365740 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:21:27.366027 (+ 287us) server_negotiation.cc:407] Connection header received
0929 03:21:27.366057 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:21:27.366058 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:21:27.366071 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:21:27.366121 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:21:27.366265 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:27.366723 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:27.367655 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:27.368230 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:27.368302 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:21:27.368310 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0929 03:21:27.368466 (+ 156us) server_negotiation.cc:299] Negotiation successful
0929 03:21:27.368503 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260929 03:21:27.368547 8328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:27.365583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.65:41855 (local address 127.20.178.67:44005)
0929 03:21:27.365636 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0929 03:21:27.365648 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 03:21:27.365716 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:21:27.365890 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:21:27.365891 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:21:27.365893 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:21:27.365970 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:21:27.365971 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:21:27.366736 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:21:27.366737 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:21:27.367525 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:21:27.367527 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:21:27.368261 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:21:27.368262 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:21:27.368349 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:21:27.368353 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 03:21:27.368402 (+ 49us) client_negotiation.cc:240] Negotiation successful
0929 03:21:27.368458 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34,"mutex_wait_us":27}
I20260929 03:21:27.368624 8349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:27.365584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.66:34733 (local address 127.20.178.67:55427)
0929 03:21:27.365926 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0929 03:21:27.365936 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 03:21:27.365994 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:21:27.366133 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:21:27.366134 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:21:27.366136 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:21:27.366198 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:21:27.366199 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:21:27.366811 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:21:27.366811 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:21:27.367539 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:21:27.367541 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:21:27.368318 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:21:27.368319 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:21:27.368371 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:21:27.368375 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 03:21:27.368451 (+ 76us) client_negotiation.cc:240] Negotiation successful
0929 03:21:27.368487 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":326,"mutex_wait_us":181,"thread_start_us":115,"threads_started":1}
I20260929 03:21:27.368927 8012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d22cf21b3fb4f5e840598f5002267a3" candidate_uuid: "89b68eb29a8644d38d3ca00ad67d532a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f276fcb82f3041219689f781013028f4" is_pre_election: true
I20260929 03:21:27.369082 8012 raft_consensus.cc:2454] T 1d22cf21b3fb4f5e840598f5002267a3 P f276fcb82f3041219689f781013028f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89b68eb29a8644d38d3ca00ad67d532a in term 0.
I20260929 03:21:27.369179 8146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d22cf21b3fb4f5e840598f5002267a3" candidate_uuid: "89b68eb29a8644d38d3ca00ad67d532a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e86cc27863094a27a902cee18e7567c2" is_pre_election: true
I20260929 03:21:27.369343 8146 raft_consensus.cc:2454] T 1d22cf21b3fb4f5e840598f5002267a3 P e86cc27863094a27a902cee18e7567c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89b68eb29a8644d38d3ca00ad67d532a in term 0.
I20260929 03:21:27.369359 8214 leader_election.cc:302] T 1d22cf21b3fb4f5e840598f5002267a3 P 89b68eb29a8644d38d3ca00ad67d532a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89b68eb29a8644d38d3ca00ad67d532a, f276fcb82f3041219689f781013028f4; no voters:
I20260929 03:21:27.369503 8347 raft_consensus.cc:2781] T 1d22cf21b3fb4f5e840598f5002267a3 P 89b68eb29a8644d38d3ca00ad67d532a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 03:21:27.369554 8347 raft_consensus.cc:482] T 1d22cf21b3fb4f5e840598f5002267a3 P 89b68eb29a8644d38d3ca00ad67d532a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 03:21:27.369587 8347 raft_consensus.cc:3037] T 1d22cf21b3fb4f5e840598f5002267a3 P 89b68eb29a8644d38d3ca00ad67d532a [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:21:27.370316 8347 raft_consensus.cc:504] T 1d22cf21b3fb4f5e840598f5002267a3 P 89b68eb29a8644d38d3ca00ad67d532a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e86cc27863094a27a902cee18e7567c2" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 34733 } } peers { permanent_uuid: "89b68eb29a8644d38d3ca00ad67d532a" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 42631 } } peers { permanent_uuid: "f276fcb82f3041219689f781013028f4" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 41855 } }
I20260929 03:21:27.370458 8347 leader_election.cc:288] T 1d22cf21b3fb4f5e840598f5002267a3 P 89b68eb29a8644d38d3ca00ad67d532a [CANDIDATE]: Term 1 election: Requested vote from peers e86cc27863094a27a902cee18e7567c2 (127.20.178.66:34733), f276fcb82f3041219689f781013028f4 (127.20.178.65:41855)
I20260929 03:21:27.370587 8146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d22cf21b3fb4f5e840598f5002267a3" candidate_uuid: "89b68eb29a8644d38d3ca00ad67d532a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e86cc27863094a27a902cee18e7567c2"
I20260929 03:21:27.370618 8012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d22cf21b3fb4f5e840598f5002267a3" candidate_uuid: "89b68eb29a8644d38d3ca00ad67d532a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f276fcb82f3041219689f781013028f4"
I20260929 03:21:27.370687 8146 raft_consensus.cc:3037] T 1d22cf21b3fb4f5e840598f5002267a3 P e86cc27863094a27a902cee18e7567c2 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:21:27.370689 8012 raft_consensus.cc:3037] T 1d22cf21b3fb4f5e840598f5002267a3 P f276fcb82f3041219689f781013028f4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:21:27.371440 8146 raft_consensus.cc:2454] T 1d22cf21b3fb4f5e840598f5002267a3 P e86cc27863094a27a902cee18e7567c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89b68eb29a8644d38d3ca00ad67d532a in term 1.
I20260929 03:21:27.371461 8012 raft_consensus.cc:2454] T 1d22cf21b3fb4f5e840598f5002267a3 P f276fcb82f3041219689f781013028f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89b68eb29a8644d38d3ca00ad67d532a in term 1.
I20260929 03:21:27.371635 8213 leader_election.cc:302] T 1d22cf21b3fb4f5e840598f5002267a3 P 89b68eb29a8644d38d3ca00ad67d532a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89b68eb29a8644d38d3ca00ad67d532a, e86cc27863094a27a902cee18e7567c2; no voters:
I20260929 03:21:27.371731 8347 raft_consensus.cc:2781] T 1d22cf21b3fb4f5e840598f5002267a3 P 89b68eb29a8644d38d3ca00ad67d532a [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:21:27.371871 8347 raft_consensus.cc:686] T 1d22cf21b3fb4f5e840598f5002267a3 P 89b68eb29a8644d38d3ca00ad67d532a [term 1 LEADER]: Becoming Leader. State: Replica: 89b68eb29a8644d38d3ca00ad67d532a, State: Running, Role: LEADER
I20260929 03:21:27.371973 8347 consensus_queue.cc:237] T 1d22cf21b3fb4f5e840598f5002267a3 P 89b68eb29a8644d38d3ca00ad67d532a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e86cc27863094a27a902cee18e7567c2" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 34733 } } peers { permanent_uuid: "89b68eb29a8644d38d3ca00ad67d532a" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 42631 } } peers { permanent_uuid: "f276fcb82f3041219689f781013028f4" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 41855 } }
I20260929 03:21:27.372849 7867 catalog_manager.cc:5495] T 1d22cf21b3fb4f5e840598f5002267a3 P 89b68eb29a8644d38d3ca00ad67d532a reported cstate change: term changed from 0 to 1, leader changed from <none> to 89b68eb29a8644d38d3ca00ad67d532a (127.20.178.67). New cstate: current_term: 1 leader_uuid: "89b68eb29a8644d38d3ca00ad67d532a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e86cc27863094a27a902cee18e7567c2" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 34733 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "89b68eb29a8644d38d3ca00ad67d532a" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 42631 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f276fcb82f3041219689f781013028f4" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 41855 } health_report { overall_health: UNKNOWN } } }
I20260929 03:21:27.401257 8339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:27.398908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55074 (local address 127.20.178.67:42631)
0929 03:21:27.398979 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0929 03:21:27.398981 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:21:27.398989 (+ 8us) server_negotiation.cc:407] Connection header received
0929 03:21:27.399034 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:21:27.399035 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:21:27.399053 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:21:27.399106 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:21:27.399235 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:27.399710 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:27.400448 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:27.400725 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:27.400786 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:21:27.400922 (+ 136us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:21:27.401023 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:21:27.401086 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0929 03:21:27.401115 (+ 29us) server_negotiation.cc:299] Negotiation successful
0929 03:21:27.401131 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 03:21:27.403084 8012 raft_consensus.cc:1261] T 1d22cf21b3fb4f5e840598f5002267a3 P f276fcb82f3041219689f781013028f4 [term 1 FOLLOWER]: Refusing update from remote peer 89b68eb29a8644d38d3ca00ad67d532a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:21:27.403085 8146 raft_consensus.cc:1261] T 1d22cf21b3fb4f5e840598f5002267a3 P e86cc27863094a27a902cee18e7567c2 [term 1 FOLLOWER]: Refusing update from remote peer 89b68eb29a8644d38d3ca00ad67d532a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:21:27.403455 8350 consensus_queue.cc:1035] T 1d22cf21b3fb4f5e840598f5002267a3 P 89b68eb29a8644d38d3ca00ad67d532a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f276fcb82f3041219689f781013028f4" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 41855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:21:27.403580 8347 consensus_queue.cc:1035] T 1d22cf21b3fb4f5e840598f5002267a3 P 89b68eb29a8644d38d3ca00ad67d532a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e86cc27863094a27a902cee18e7567c2" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 34733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260929 03:21:27.404861 8059 tablet.cc:2369] T 1d22cf21b3fb4f5e840598f5002267a3 P f276fcb82f3041219689f781013028f4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 03:21:27.406347 8355 mvcc.cc:205] Tried to move back new op lower bound from 7334510950000173056 to 7334510949876326400. Current Snapshot: MvccSnapshot[applied={T|T < 7334510950000173056}]
I20260929 03:21:27.408911 7867 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55296:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:21:27.409018 7867 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55296:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:21:27.409777 7867 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5ee2ea0c4da841fba478da0624532611: Sending DeleteTablet for 3 replicas of tablet 1d22cf21b3fb4f5e840598f5002267a3
I20260929 03:21:27.410099 8126 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d22cf21b3fb4f5e840598f5002267a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:21:27 UTC) from {username='kudu', principal='kudu/127.20.178.126@KRBTEST.COM'} at 127.0.0.1:34112
I20260929 03:21:27.410135 7992 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d22cf21b3fb4f5e840598f5002267a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:21:27 UTC) from {username='kudu', principal='kudu/127.20.178.126@KRBTEST.COM'} at 127.0.0.1:47246
I20260929 03:21:27.410145 8260 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d22cf21b3fb4f5e840598f5002267a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:21:27 UTC) from {username='kudu', principal='kudu/127.20.178.126@KRBTEST.COM'} at 127.0.0.1:55070
I20260929 03:21:27.410363 8365 tablet_replica.cc:330] T 1d22cf21b3fb4f5e840598f5002267a3 P 89b68eb29a8644d38d3ca00ad67d532a: stopping tablet replica
I20260929 03:21:27.410436 8365 raft_consensus.cc:2229] T 1d22cf21b3fb4f5e840598f5002267a3 P 89b68eb29a8644d38d3ca00ad67d532a [term 1 LEADER]: Raft consensus shutting down.
I20260929 03:21:27.410522 8365 raft_consensus.cc:2258] T 1d22cf21b3fb4f5e840598f5002267a3 P 89b68eb29a8644d38d3ca00ad67d532a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:21:27.410763 21193 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuoaHbX/build/release/bin/kudu with pid 7929
I20260929 03:21:27.410940 8365 ts_tablet_manager.cc:1905] T 1d22cf21b3fb4f5e840598f5002267a3 P 89b68eb29a8644d38d3ca00ad67d532a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:21:27.413064 8365 ts_tablet_manager.cc:1918] T 1d22cf21b3fb4f5e840598f5002267a3 P 89b68eb29a8644d38d3ca00ad67d532a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:21:27.413133 8365 log.cc:1196] T 1d22cf21b3fb4f5e840598f5002267a3 P 89b68eb29a8644d38d3ca00ad67d532a: Deleting WAL directory at /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651852476208-21193-0/minicluster-data/ts-2/wal/wals/1d22cf21b3fb4f5e840598f5002267a3
I20260929 03:21:27.413434 8365 ts_tablet_manager.cc:1939] T 1d22cf21b3fb4f5e840598f5002267a3 P 89b68eb29a8644d38d3ca00ad67d532a: Deleting consensus metadata
I20260929 03:21:27.413743 8363 tablet_replica.cc:330] T 1d22cf21b3fb4f5e840598f5002267a3 P e86cc27863094a27a902cee18e7567c2: stopping tablet replica
I20260929 03:21:27.413851 8363 raft_consensus.cc:2229] T 1d22cf21b3fb4f5e840598f5002267a3 P e86cc27863094a27a902cee18e7567c2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:21:27.413863 7851 catalog_manager.cc:4841] TS 89b68eb29a8644d38d3ca00ad67d532a (127.20.178.67:42631): tablet 1d22cf21b3fb4f5e840598f5002267a3 (table test-table [id=3745aa5c4a084ee4acaad65ee39f443b]) successfully deleted
I20260929 03:21:27.413954 8363 raft_consensus.cc:2258] T 1d22cf21b3fb4f5e840598f5002267a3 P e86cc27863094a27a902cee18e7567c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:21:27.414402 8363 ts_tablet_manager.cc:1905] T 1d22cf21b3fb4f5e840598f5002267a3 P e86cc27863094a27a902cee18e7567c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:21:27.416323 8363 ts_tablet_manager.cc:1918] T 1d22cf21b3fb4f5e840598f5002267a3 P e86cc27863094a27a902cee18e7567c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:21:27.416447 8363 log.cc:1196] T 1d22cf21b3fb4f5e840598f5002267a3 P e86cc27863094a27a902cee18e7567c2: Deleting WAL directory at /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651852476208-21193-0/minicluster-data/ts-1/wal/wals/1d22cf21b3fb4f5e840598f5002267a3
I20260929 03:21:27.416795 8363 ts_tablet_manager.cc:1939] T 1d22cf21b3fb4f5e840598f5002267a3 P e86cc27863094a27a902cee18e7567c2: Deleting consensus metadata
I20260929 03:21:27.417217 7851 catalog_manager.cc:4841] TS e86cc27863094a27a902cee18e7567c2 (127.20.178.66:34733): tablet 1d22cf21b3fb4f5e840598f5002267a3 (table test-table [id=3745aa5c4a084ee4acaad65ee39f443b]) successfully deleted
W20260929 03:21:27.417980 7852 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.20.178.65:41855 (error 108)
I20260929 03:21:27.418481 8335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:27.418360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.65:41855 (local address 127.0.0.1:47262)
0929 03:21:27.418409 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0929 03:21:27.418443 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.178.65:41855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260929 03:21:27.418550 21193 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuoaHbX/build/release/bin/kudu with pid 8063
W20260929 03:21:27.418591 7852 catalog_manager.cc:4584] TS f276fcb82f3041219689f781013028f4 (127.20.178.65:41855): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1d22cf21b3fb4f5e840598f5002267a3: Network error: Client connection negotiation failed: client connection to 127.20.178.65:41855: connect: Connection refused (error 111)
I20260929 03:21:27.424077 21193 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuoaHbX/build/release/bin/kudu with pid 8197
I20260929 03:21:27.428399 21193 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuoaHbX/build/release/bin/kudu with pid 7835
2026-09-29T03:21:27Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (1954 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-29T03:21:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:21:27Z Disabled control of system clock
I20260929 03:21:27.456411 21193 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuoaHbX/build/release/bin/kudu
/tmp/dist-test-taskuoaHbX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790651852476208-21193-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790651852476208-21193-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790651852476208-21193-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790651852476208-21193-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.178.126:33261
--webserver_interface=127.20.178.126
--webserver_port=0
--builtin_ntp_servers=127.20.178.84:37003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.178.126:33261
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 03:21:27.537856 8370 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:21:27.538102 8370 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:21:27.538138 8370 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:21:27.539280 8370 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:21:27.539347 8370 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:21:27.539368 8370 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:21:27.539389 8370 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:21:27.539417 8370 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:21:27.541375 8370 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.178.84:37003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790651852476208-21193-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790651852476208-21193-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.178.126:33261
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.178.126:33261
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790651852476208-21193-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.178.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8370
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790651852476208-21193-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:21:27.541663 8370 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:21:27.541965 8370 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:21:27.546885 8376 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:21:27.546938 8370 server_base.cc:1034] running on GCE node
W20260929 03:21:27.546914 8375 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:21:27.546914 8378 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:21:27.547444 8370 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:21:27.547744 8370 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:21:27.548888 8370 hybrid_clock.cc:648] HybridClock initialized: now 1790652087548878 us; error 29 us; skew 500 ppm
I20260929 03:21:27.557333 8370 webserver.cc:460] Webserver started at http://127.20.178.126:46553/ using document root <none> and password file <none>
I20260929 03:21:27.557628 8370 fs_manager.cc:359] Metadata directory not provided
I20260929 03:21:27.557678 8370 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:21:27.557837 8370 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:21:27.558916 8370 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790651852476208-21193-0/minicluster-data/master-0/data/instance:
uuid: "fec78c577ee147fba89b0a315218406a"
format_stamp: "Formatted at 2026-09-29 03:21:27 on dist-test-slave-c9qh"
server_key: "c674cd018b1108930a287968f19a9198"
server_key_iv: "d0ae1c07a0c1c1546153772d7790008b"
server_key_version: "encryptionkey@0"
I20260929 03:21:27.559278 8370 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790651852476208-21193-0/minicluster-data/master-0/wal/instance:
uuid: "fec78c577ee147fba89b0a315218406a"
format_stamp: "Formatted at 2026-09-29 03:21:27 on dist-test-slave-c9qh"
server_key: "c674cd018b1108930a287968f19a9198"
server_key_iv: "d0ae1c07a0c1c1546153772d7790008b"
server_key_version: "encryptionkey@0"
I20260929 03:21:27.560578 8370 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 03:21:27.561386 8384 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:21:27.561574 8370 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260929 03:21:27.561667 8370 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790651852476208-21193-0/minicluster-data/master-0/data,/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790651852476208-21193-0/minicluster-data/master-0/wal
uuid: "fec78c577ee147fba89b0a315218406a"
format_stamp: "Formatted at 2026-09-29 03:21:27 on dist-test-slave-c9qh"
server_key: "c674cd018b1108930a287968f19a9198"
server_key_iv: "d0ae1c07a0c1c1546153772d7790008b"
server_key_version: "encryptionkey@0"
I20260929 03:21:27.561741 8370 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790651852476208-21193-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790651852476208-21193-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790651852476208-21193-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:21:27.572244 8370 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:21:27.572543 8370 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:21:27.572722 8370 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:21:27.576548 8370 rpc_server.cc:307] RPC server started. Bound to: 127.20.178.126:33261
I20260929 03:21:27.576579 8436 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.178.126:33261 every 8 connection(s)
I20260929 03:21:27.576942 8370 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790651852476208-21193-0/minicluster-data/master-0/data/info.pb
I20260929 03:21:27.577857 8437 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:21:27.580101 8437 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fec78c577ee147fba89b0a315218406a: Bootstrap starting.
I20260929 03:21:27.580749 8437 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fec78c577ee147fba89b0a315218406a: Neither blocks nor log segments found. Creating new log.
I20260929 03:21:27.581086 21193 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuoaHbX/build/release/bin/kudu as pid 8370
I20260929 03:21:27.581089 8437 log.cc:824] T 00000000000000000000000000000000 P fec78c577ee147fba89b0a315218406a: Log is configured to *not* fsync() on all Append() calls
I20260929 03:21:27.581189 21193 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790651852476208-21193-0/minicluster-data/master-0/wal/instance
I20260929 03:21:27.581388 21193 external_mini_cluster.cc:1428] Setting key ec5ee72ba13b22b920025342dbb0bbb2
I20260929 03:21:27.582062 8437 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fec78c577ee147fba89b0a315218406a: No bootstrap required, opened a new log
I20260929 03:21:27.584111 8437 raft_consensus.cc:348] T 00000000000000000000000000000000 P fec78c577ee147fba89b0a315218406a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fec78c577ee147fba89b0a315218406a" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 33261 } }
I20260929 03:21:27.584272 8437 raft_consensus.cc:374] T 00000000000000000000000000000000 P fec78c577ee147fba89b0a315218406a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:21:27.584321 8437 raft_consensus.cc:729] T 00000000000000000000000000000000 P fec78c577ee147fba89b0a315218406a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fec78c577ee147fba89b0a315218406a, State: Initialized, Role: FOLLOWER
I20260929 03:21:27.584465 8437 consensus_queue.cc:260] T 00000000000000000000000000000000 P fec78c577ee147fba89b0a315218406a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fec78c577ee147fba89b0a315218406a" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 33261 } }
I20260929 03:21:27.584563 8437 raft_consensus.cc:388] T 00000000000000000000000000000000 P fec78c577ee147fba89b0a315218406a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:21:27.584605 8437 raft_consensus.cc:482] T 00000000000000000000000000000000 P fec78c577ee147fba89b0a315218406a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:21:27.584676 8437 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fec78c577ee147fba89b0a315218406a [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:21:27.585219 8437 raft_consensus.cc:504] T 00000000000000000000000000000000 P fec78c577ee147fba89b0a315218406a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fec78c577ee147fba89b0a315218406a" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 33261 } }
I20260929 03:21:27.585340 8437 leader_election.cc:302] T 00000000000000000000000000000000 P fec78c577ee147fba89b0a315218406a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fec78c577ee147fba89b0a315218406a; no voters:
I20260929 03:21:27.585496 8437 leader_election.cc:288] T 00000000000000000000000000000000 P fec78c577ee147fba89b0a315218406a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:21:27.585526 8442 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fec78c577ee147fba89b0a315218406a [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:21:27.585644 8442 raft_consensus.cc:686] T 00000000000000000000000000000000 P fec78c577ee147fba89b0a315218406a [term 1 LEADER]: Becoming Leader. State: Replica: fec78c577ee147fba89b0a315218406a, State: Running, Role: LEADER
I20260929 03:21:27.585727 8442 consensus_queue.cc:237] T 00000000000000000000000000000000 P fec78c577ee147fba89b0a315218406a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fec78c577ee147fba89b0a315218406a" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 33261 } }
I20260929 03:21:27.586045 8444 sys_catalog.cc:455] T 00000000000000000000000000000000 P fec78c577ee147fba89b0a315218406a [sys.catalog]: SysCatalogTable state changed. Reason: New leader fec78c577ee147fba89b0a315218406a. Latest consensus state: current_term: 1 leader_uuid: "fec78c577ee147fba89b0a315218406a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fec78c577ee147fba89b0a315218406a" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 33261 } } }
I20260929 03:21:27.586108 8443 sys_catalog.cc:455] T 00000000000000000000000000000000 P fec78c577ee147fba89b0a315218406a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fec78c577ee147fba89b0a315218406a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fec78c577ee147fba89b0a315218406a" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 33261 } } }
I20260929 03:21:27.586215 8444 sys_catalog.cc:458] T 00000000000000000000000000000000 P fec78c577ee147fba89b0a315218406a [sys.catalog]: This master's current role is: LEADER
I20260929 03:21:27.586232 8443 sys_catalog.cc:458] T 00000000000000000000000000000000 P fec78c577ee147fba89b0a315218406a [sys.catalog]: This master's current role is: LEADER
I20260929 03:21:27.586149 8437 sys_catalog.cc:564] T 00000000000000000000000000000000 P fec78c577ee147fba89b0a315218406a [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:21:27.586464 8446 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:21:27.586979 8446 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:21:27.587059 8441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:27.581875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48908 (local address 127.20.178.126:33261)
0929 03:21:27.582079 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0929 03:21:27.582088 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 03:21:27.582118 (+ 30us) server_negotiation.cc:407] Connection header received
0929 03:21:27.582558 (+ 440us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:21:27.582564 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:21:27.582621 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:21:27.582763 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:21:27.582994 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:27.583868 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:27.584804 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:27.585215 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:27.585360 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:21:27.586098 (+ 738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:21:27.586101 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:21:27.586102 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:21:27.586311 (+ 209us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:21:27.586364 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:21:27.586380 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:21:27.586457 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0929 03:21:27.586717 (+ 260us) server_negotiation.cc:299] Negotiation successful
0929 03:21:27.586809 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":71,"threads_started":1}
I20260929 03:21:27.588712 8446 catalog_manager.cc:1347] Generated new cluster ID: a8123264ae2845e486e40f06e7398fa4
I20260929 03:21:27.588760 8446 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:21:27.611797 8446 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:21:27.612357 8446 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:21:27.621021 8446 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fec78c577ee147fba89b0a315218406a: Generated new TSK 0
I20260929 03:21:27.621228 8446 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 03:21:27.623559 21193 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuoaHbX/build/release/bin/kudu
/tmp/dist-test-taskuoaHbX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790651852476208-21193-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790651852476208-21193-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790651852476208-21193-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790651852476208-21193-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.178.65:0
--local_ip_for_outbound_sockets=127.20.178.65
--webserver_interface=127.20.178.65
--webserver_port=0
--tserver_master_addrs=127.20.178.126:33261
--builtin_ntp_servers=127.20.178.84:37003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 03:21:27.718317 8461 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:21:27.718621 8461 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:21:27.718655 8461 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:21:27.720062 8461 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:21:27.720130 8461 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:21:27.720176 8461 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.178.65
I20260929 03:21:27.721622 8461 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.178.84:37003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790651852476208-21193-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790651852476208-21193-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.178.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790651852476208-21193-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.178.65
--webserver_port=0
--tserver_master_addrs=127.20.178.126:33261
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8461
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.178.65
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790651852476208-21193-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:21:27.721918 8461 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:21:27.722265 8461 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:21:27.728020 8469 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:21:27.728022 8467 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:21:27.728298 8466 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:21:27.733964 8461 server_base.cc:1034] running on GCE node
I20260929 03:21:27.734197 8461 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:21:27.734581 8461 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:21:27.735725 8461 hybrid_clock.cc:648] HybridClock initialized: now 1790652087735710 us; error 38 us; skew 500 ppm
I20260929 03:21:27.737188 8461 webserver.cc:460] Webserver started at http://127.20.178.65:39949/ using document root <none> and password file <none>
I20260929 03:21:27.737542 8461 fs_manager.cc:359] Metadata directory not provided
I20260929 03:21:27.737607 8461 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:21:27.737743 8461 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:21:27.739058 8461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790651852476208-21193-0/minicluster-data/ts-0/data/instance:
uuid: "94305630600f48dbb68c3af48fd41bc4"
format_stamp: "Formatted at 2026-09-29 03:21:27 on dist-test-slave-c9qh"
server_key: "69718266332f6a8df5ee21d30debf415"
server_key_iv: "c6faee07a70426e8324096ff8e0a7c0a"
server_key_version: "encryptionkey@0"
I20260929 03:21:27.739446 8461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790651852476208-21193-0/minicluster-data/ts-0/wal/instance:
uuid: "94305630600f48dbb68c3af48fd41bc4"
format_stamp: "Formatted at 2026-09-29 03:21:27 on dist-test-slave-c9qh"
server_key: "69718266332f6a8df5ee21d30debf415"
server_key_iv: "c6faee07a70426e8324096ff8e0a7c0a"
server_key_version: "encryptionkey@0"
I20260929 03:21:27.741221 8461 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.002s
I20260929 03:21:27.742039 8475 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:21:27.742203 8461 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 03:21:27.742295 8461 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790651852476208-21193-0/minicluster-data/ts-0/data,/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790651852476208-21193-0/minicluster-data/ts-0/wal
uuid: "94305630600f48dbb68c3af48fd41bc4"
format_stamp: "Formatted at 2026-09-29 03:21:27 on dist-test-slave-c9qh"
server_key: "69718266332f6a8df5ee21d30debf415"
server_key_iv: "c6faee07a70426e8324096ff8e0a7c0a"
server_key_version: "encryptionkey@0"
I20260929 03:21:27.742388 8461 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790651852476208-21193-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790651852476208-21193-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790651852476208-21193-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:21:27.759450 8461 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:21:27.759769 8461 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:21:27.759920 8461 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:21:27.760152 8461 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:21:27.760493 8461 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:21:27.760551 8461 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:21:27.760593 8461 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:21:27.760622 8461 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:21:27.766522 8461 rpc_server.cc:307] RPC server started. Bound to: 127.20.178.65:44255
I20260929 03:21:27.766600 8588 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.178.65:44255 every 8 connection(s)
I20260929 03:21:27.766928 8461 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790651852476208-21193-0/minicluster-data/ts-0/data/info.pb
I20260929 03:21:27.769315 21193 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuoaHbX/build/release/bin/kudu as pid 8461
I20260929 03:21:27.769431 21193 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790651852476208-21193-0/minicluster-data/ts-0/wal/instance
I20260929 03:21:27.769599 21193 external_mini_cluster.cc:1428] Setting key 435ba84c190540a7dfc40bf927c1de3f
I20260929 03:21:27.770502 21193 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuoaHbX/build/release/bin/kudu
/tmp/dist-test-taskuoaHbX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790651852476208-21193-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790651852476208-21193-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790651852476208-21193-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790651852476208-21193-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.178.66:0
--local_ip_for_outbound_sockets=127.20.178.66
--webserver_interface=127.20.178.66
--webserver_port=0
--tserver_master_addrs=127.20.178.126:33261
--builtin_ntp_servers=127.20.178.84:37003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 03:21:27.772339 8441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:27.767693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.65:55219 (local address 127.20.178.126:33261)
0929 03:21:27.767738 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0929 03:21:27.767740 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:21:27.767984 (+ 244us) server_negotiation.cc:407] Connection header received
0929 03:21:27.768447 (+ 463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:21:27.768448 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:21:27.768458 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:21:27.768510 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:21:27.769251 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:27.769888 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:27.770880 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:27.771094 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:27.771183 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:21:27.772068 (+ 885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:21:27.772070 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:21:27.772070 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:21:27.772075 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:21:27.772090 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:21:27.772094 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:21:27.772141 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0929 03:21:27.772233 (+ 92us) server_negotiation.cc:299] Negotiation successful
0929 03:21:27.772258 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260929 03:21:27.772581 8591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:27.767676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:33261 (local address 127.20.178.65:55219)
0929 03:21:27.767855 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0929 03:21:27.767889 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0929 03:21:27.768363 (+ 474us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:21:27.768565 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:21:27.768568 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:21:27.768577 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:21:27.769149 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:21:27.769155 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:21:27.769990 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:21:27.769993 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:21:27.770782 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:21:27.770784 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:21:27.771195 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:21:27.771196 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:21:27.771328 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:21:27.771910 (+ 582us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 03:21:27.771910 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:21:27.771939 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 03:21:27.771961 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:21:27.772141 (+ 180us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:21:27.772146 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:21:27.772150 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 03:21:27.772235 (+ 85us) client_negotiation.cc:240] Negotiation successful
0929 03:21:27.772335 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":92,"threads_started":1}
I20260929 03:21:27.773123 8589 heartbeater.cc:347] Connected to a master server at 127.20.178.126:33261
I20260929 03:21:27.773234 8589 heartbeater.cc:464] Registering TS with master...
I20260929 03:21:27.773416 8589 heartbeater.cc:511] Master 127.20.178.126:33261 requested a full tablet report, sending...
I20260929 03:21:27.773933 8401 ts_manager.cc:194] Registered new tserver with Master: 94305630600f48dbb68c3af48fd41bc4 (127.20.178.65:44255)
I20260929 03:21:27.774935 8401 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.178.65:55219
W20260929 03:21:27.854360 8592 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:21:27.854557 8592 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:21:27.854578 8592 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:21:27.855634 8592 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:21:27.855665 8592 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:21:27.855690 8592 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.178.66
I20260929 03:21:27.856994 8592 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.178.84:37003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790651852476208-21193-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790651852476208-21193-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.178.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790651852476208-21193-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.178.66
--webserver_port=0
--tserver_master_addrs=127.20.178.126:33261
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8592
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.178.66
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790651852476208-21193-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:21:27.857177 8592 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:21:27.857400 8592 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:21:27.862330 8597 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:21:27.862360 8600 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:21:27.862342 8598 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:21:27.862617 8592 server_base.cc:1034] running on GCE node
I20260929 03:21:27.862845 8592 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:21:27.863102 8592 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:21:27.864251 8592 hybrid_clock.cc:648] HybridClock initialized: now 1790652087864229 us; error 44 us; skew 500 ppm
I20260929 03:21:27.865685 8592 webserver.cc:460] Webserver started at http://127.20.178.66:34293/ using document root <none> and password file <none>
I20260929 03:21:27.865886 8592 fs_manager.cc:359] Metadata directory not provided
I20260929 03:21:27.865929 8592 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:21:27.866020 8592 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:21:27.867101 8592 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790651852476208-21193-0/minicluster-data/ts-1/data/instance:
uuid: "d610a77bd83c43bf9f11721fb5990a9d"
format_stamp: "Formatted at 2026-09-29 03:21:27 on dist-test-slave-c9qh"
server_key: "ca8a2d8b4a814b6b8c433a81213d9c87"
server_key_iv: "91d047763ea5cf4d94d44e099e2baaa5"
server_key_version: "encryptionkey@0"
I20260929 03:21:27.867409 8592 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790651852476208-21193-0/minicluster-data/ts-1/wal/instance:
uuid: "d610a77bd83c43bf9f11721fb5990a9d"
format_stamp: "Formatted at 2026-09-29 03:21:27 on dist-test-slave-c9qh"
server_key: "ca8a2d8b4a814b6b8c433a81213d9c87"
server_key_iv: "91d047763ea5cf4d94d44e099e2baaa5"
server_key_version: "encryptionkey@0"
I20260929 03:21:27.868608 8592 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 03:21:27.869367 8606 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:21:27.869549 8592 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 03:21:27.869647 8592 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790651852476208-21193-0/minicluster-data/ts-1/data,/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790651852476208-21193-0/minicluster-data/ts-1/wal
uuid: "d610a77bd83c43bf9f11721fb5990a9d"
format_stamp: "Formatted at 2026-09-29 03:21:27 on dist-test-slave-c9qh"
server_key: "ca8a2d8b4a814b6b8c433a81213d9c87"
server_key_iv: "91d047763ea5cf4d94d44e099e2baaa5"
server_key_version: "encryptionkey@0"
I20260929 03:21:27.869740 8592 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790651852476208-21193-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790651852476208-21193-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790651852476208-21193-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:21:27.877609 8592 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:21:27.877898 8592 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:21:27.878044 8592 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:21:27.878259 8592 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:21:27.878582 8592 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:21:27.878636 8592 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:21:27.878676 8592 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:21:27.878705 8592 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:21:27.884353 8592 rpc_server.cc:307] RPC server started. Bound to: 127.20.178.66:40079
I20260929 03:21:27.884418 8719 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.178.66:40079 every 8 connection(s)
I20260929 03:21:27.884814 8592 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790651852476208-21193-0/minicluster-data/ts-1/data/info.pb
I20260929 03:21:27.885514 21193 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuoaHbX/build/release/bin/kudu as pid 8592
I20260929 03:21:27.885627 21193 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790651852476208-21193-0/minicluster-data/ts-1/wal/instance
I20260929 03:21:27.885826 21193 external_mini_cluster.cc:1428] Setting key e0a007a160ab6141a66910ab0b17b6ad
I20260929 03:21:27.886904 21193 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuoaHbX/build/release/bin/kudu
/tmp/dist-test-taskuoaHbX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790651852476208-21193-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790651852476208-21193-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790651852476208-21193-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790651852476208-21193-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.178.67:0
--local_ip_for_outbound_sockets=127.20.178.67
--webserver_interface=127.20.178.67
--webserver_port=0
--tserver_master_addrs=127.20.178.126:33261
--builtin_ntp_servers=127.20.178.84:37003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 03:21:27.891175 8441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:27.885571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.66:41619 (local address 127.20.178.126:33261)
0929 03:21:27.885627 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0929 03:21:27.885629 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:21:27.885864 (+ 235us) server_negotiation.cc:407] Connection header received
0929 03:21:27.886434 (+ 570us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:21:27.886435 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:21:27.886446 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:21:27.886505 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:21:27.887358 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:27.888158 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:27.889716 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:27.889911 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:27.889969 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:21:27.890923 (+ 954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:21:27.890924 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:21:27.890924 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:21:27.890928 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:21:27.890942 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:21:27.890946 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:21:27.890989 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0929 03:21:27.891097 (+ 108us) server_negotiation.cc:299] Negotiation successful
0929 03:21:27.891117 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260929 03:21:27.891494 8722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:27.885522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:33261 (local address 127.20.178.66:41619)
0929 03:21:27.885723 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0929 03:21:27.885763 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 03:21:27.886330 (+ 567us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:21:27.886580 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:21:27.886584 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:21:27.886593 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:21:27.887234 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:21:27.887258 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:21:27.888271 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:21:27.888273 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:21:27.889623 (+ 1350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:21:27.889626 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:21:27.889999 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:21:27.890000 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:21:27.890161 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:21:27.890760 (+ 599us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 03:21:27.890760 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:21:27.890802 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 03:21:27.890833 (+ 31us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:21:27.891013 (+ 180us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:21:27.891017 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:21:27.891020 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 03:21:27.891096 (+ 76us) client_negotiation.cc:240] Negotiation successful
0929 03:21:27.891214 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":96,"threads_started":1}
I20260929 03:21:27.891978 8720 heartbeater.cc:347] Connected to a master server at 127.20.178.126:33261
I20260929 03:21:27.892082 8720 heartbeater.cc:464] Registering TS with master...
I20260929 03:21:27.892261 8720 heartbeater.cc:511] Master 127.20.178.126:33261 requested a full tablet report, sending...
I20260929 03:21:27.892689 8401 ts_manager.cc:194] Registered new tserver with Master: d610a77bd83c43bf9f11721fb5990a9d (127.20.178.66:40079)
I20260929 03:21:27.893079 8401 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.178.66:41619
W20260929 03:21:27.967099 8723 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:21:27.967301 8723 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:21:27.967324 8723 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:21:27.968484 8723 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:21:27.968524 8723 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:21:27.968549 8723 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.178.67
I20260929 03:21:27.969950 8723 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.178.84:37003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790651852476208-21193-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790651852476208-21193-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.178.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790651852476208-21193-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.178.67
--webserver_port=0
--tserver_master_addrs=127.20.178.126:33261
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8723
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.178.67
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790651852476208-21193-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:21:27.970214 8723 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:21:27.970520 8723 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:21:27.975615 8731 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:21:27.975625 8728 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:21:27.975625 8729 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:21:27.975785 8723 server_base.cc:1034] running on GCE node
I20260929 03:21:27.976063 8723 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:21:27.976306 8723 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:21:27.977468 8723 hybrid_clock.cc:648] HybridClock initialized: now 1790652087977460 us; error 49 us; skew 500 ppm
I20260929 03:21:27.978672 8723 webserver.cc:460] Webserver started at http://127.20.178.67:41417/ using document root <none> and password file <none>
I20260929 03:21:27.978942 8723 fs_manager.cc:359] Metadata directory not provided
I20260929 03:21:27.979009 8723 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:21:27.979156 8723 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:21:27.980217 8723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790651852476208-21193-0/minicluster-data/ts-2/data/instance:
uuid: "d311e7150f1b468da7512081f03958e7"
format_stamp: "Formatted at 2026-09-29 03:21:27 on dist-test-slave-c9qh"
server_key: "1d48f43828d64e8cd978c9e9d0b7b09c"
server_key_iv: "04281f7b085e5c1dc4789daa642ad785"
server_key_version: "encryptionkey@0"
I20260929 03:21:27.980584 8723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790651852476208-21193-0/minicluster-data/ts-2/wal/instance:
uuid: "d311e7150f1b468da7512081f03958e7"
format_stamp: "Formatted at 2026-09-29 03:21:27 on dist-test-slave-c9qh"
server_key: "1d48f43828d64e8cd978c9e9d0b7b09c"
server_key_iv: "04281f7b085e5c1dc4789daa642ad785"
server_key_version: "encryptionkey@0"
I20260929 03:21:27.981876 8723 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 03:21:27.982568 8737 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:21:27.982730 8723 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 03:21:27.982823 8723 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790651852476208-21193-0/minicluster-data/ts-2/data,/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790651852476208-21193-0/minicluster-data/ts-2/wal
uuid: "d311e7150f1b468da7512081f03958e7"
format_stamp: "Formatted at 2026-09-29 03:21:27 on dist-test-slave-c9qh"
server_key: "1d48f43828d64e8cd978c9e9d0b7b09c"
server_key_iv: "04281f7b085e5c1dc4789daa642ad785"
server_key_version: "encryptionkey@0"
I20260929 03:21:27.982908 8723 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790651852476208-21193-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790651852476208-21193-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790651852476208-21193-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:21:27.999567 8723 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:21:27.999878 8723 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:21:28.000026 8723 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:21:28.000254 8723 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:21:28.000592 8723 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:21:28.000665 8723 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:21:28.000707 8723 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:21:28.000741 8723 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:21:28.006476 8723 rpc_server.cc:307] RPC server started. Bound to: 127.20.178.67:45555
I20260929 03:21:28.006523 8850 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.178.67:45555 every 8 connection(s)
I20260929 03:21:28.006855 8723 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790651852476208-21193-0/minicluster-data/ts-2/data/info.pb
I20260929 03:21:28.011796 8441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:28.007574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.67:53217 (local address 127.20.178.126:33261)
0929 03:21:28.007639 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0929 03:21:28.007641 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:21:28.007851 (+ 210us) server_negotiation.cc:407] Connection header received
0929 03:21:28.008316 (+ 465us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:21:28.008317 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:21:28.008326 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:21:28.008398 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:21:28.009134 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:28.009614 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:28.010541 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:28.010721 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:28.010772 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:21:28.011585 (+ 813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:21:28.011585 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:21:28.011586 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:21:28.011590 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:21:28.011600 (+ 10us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:21:28.011603 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:21:28.011640 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0929 03:21:28.011719 (+ 79us) server_negotiation.cc:299] Negotiation successful
0929 03:21:28.011738 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260929 03:21:28.012051 8853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:28.007535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:33261 (local address 127.20.178.67:53217)
0929 03:21:28.007719 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0929 03:21:28.007760 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 03:21:28.008236 (+ 476us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:21:28.008473 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:21:28.008476 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:21:28.008484 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:21:28.009058 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:21:28.009064 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:21:28.009691 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:21:28.009691 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:21:28.010468 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:21:28.010470 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:21:28.010791 (+ 321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:21:28.010792 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:21:28.010903 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:21:28.011464 (+ 561us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 03:21:28.011464 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:21:28.011493 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 03:21:28.011515 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:21:28.011654 (+ 139us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:21:28.011658 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:21:28.011661 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 03:21:28.011711 (+ 50us) client_negotiation.cc:240] Negotiation successful
0929 03:21:28.011788 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":91,"threads_started":1}
I20260929 03:21:28.012209 21193 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuoaHbX/build/release/bin/kudu as pid 8723
I20260929 03:21:28.012320 21193 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790651852476208-21193-0/minicluster-data/ts-2/wal/instance
I20260929 03:21:28.012532 21193 external_mini_cluster.cc:1428] Setting key 3762de1202fc64a6f352e3c3fa9d9ab6
I20260929 03:21:28.012717 8851 heartbeater.cc:347] Connected to a master server at 127.20.178.126:33261
I20260929 03:21:28.012830 8851 heartbeater.cc:464] Registering TS with master...
I20260929 03:21:28.013044 8851 heartbeater.cc:511] Master 127.20.178.126:33261 requested a full tablet report, sending...
I20260929 03:21:28.013525 8401 ts_manager.cc:194] Registered new tserver with Master: d311e7150f1b468da7512081f03958e7 (127.20.178.67:45555)
I20260929 03:21:28.013899 8401 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.178.67:53217
I20260929 03:21:28.024405 21193 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 03:21:28.028684 8441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:28.025864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48912 (local address 127.20.178.126:33261)
0929 03:21:28.025914 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0929 03:21:28.025916 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:21:28.026031 (+ 115us) server_negotiation.cc:407] Connection header received
0929 03:21:28.026091 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:21:28.026091 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:21:28.026100 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:21:28.026148 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:21:28.026283 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:28.026823 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:28.027530 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:28.027706 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:28.027756 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:21:28.028432 (+ 676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:21:28.028433 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:21:28.028433 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:21:28.028437 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:21:28.028448 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:21:28.028451 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:21:28.028489 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0929 03:21:28.028541 (+ 52us) server_negotiation.cc:299] Negotiation successful
0929 03:21:28.028558 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260929 03:21:28.030511 8401 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48912:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 03:21:28.032060 8401 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 03:21:28.038705 8861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:28.034401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34654 (local address 127.20.178.65:44255)
0929 03:21:28.034878 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0929 03:21:28.034883 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:21:28.034893 (+ 10us) server_negotiation.cc:407] Connection header received
0929 03:21:28.034933 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:21:28.034936 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:21:28.034950 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:21:28.035051 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:21:28.035252 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:28.036423 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:28.037624 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:28.038312 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:28.038403 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:21:28.038497 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0929 03:21:28.038574 (+ 77us) server_negotiation.cc:299] Negotiation successful
0929 03:21:28.038619 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":437,"thread_start_us":93,"threads_started":1}
I20260929 03:21:28.038723 8860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:28.034311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.65:44255 (local address 127.0.0.1:34654)
0929 03:21:28.034467 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0929 03:21:28.034509 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 03:21:28.034644 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:21:28.035059 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:21:28.035063 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:21:28.035065 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:21:28.035128 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:21:28.035130 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:21:28.036532 (+ 1402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:21:28.036533 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:21:28.037537 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:21:28.037539 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:21:28.038411 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:21:28.038412 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:21:28.038472 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:21:28.038484 (+ 12us) client_negotiation.cc:769] Sending connection context
0929 03:21:28.038557 (+ 73us) client_negotiation.cc:240] Negotiation successful
0929 03:21:28.038629 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"mutex_wait_us":2,"thread_start_us":99,"threads_started":1}
I20260929 03:21:28.039254 8862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:28.034532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.67:45555 (local address 127.0.0.1:54666)
0929 03:21:28.034721 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0929 03:21:28.034730 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 03:21:28.034764 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:21:28.035208 (+ 444us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:21:28.035208 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:21:28.035211 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:21:28.035254 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:21:28.035255 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:21:28.036506 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:21:28.036507 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:21:28.037755 (+ 1248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:21:28.037757 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:21:28.038636 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:21:28.038637 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:21:28.038738 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:21:28.038743 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 03:21:28.039137 (+ 394us) client_negotiation.cc:240] Negotiation successful
0929 03:21:28.039183 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"mutex_wait_us":4,"thread_start_us":64,"threads_started":1}
I20260929 03:21:28.039278 8863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:28.034634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54666 (local address 127.20.178.67:45555)
0929 03:21:28.034837 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0929 03:21:28.034841 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:21:28.034850 (+ 9us) server_negotiation.cc:407] Connection header received
0929 03:21:28.034911 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:21:28.034913 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:21:28.034931 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:21:28.035036 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:21:28.035335 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:28.036418 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:28.037864 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:28.038551 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:28.038621 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:21:28.038700 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0929 03:21:28.039159 (+ 459us) server_negotiation.cc:299] Negotiation successful
0929 03:21:28.039197 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":88,"threads_started":1}
I20260929 03:21:28.039569 8523 tablet_service.cc:1467] Processing CreateTablet for tablet 4b02b1c2ffd1414a9e1fee0f037551b3 (DEFAULT_TABLE table=test-table [id=88306341fa1743a683df522de23d249e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:21:28.040028 8523 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b02b1c2ffd1414a9e1fee0f037551b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:21:28.040050 8859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:28.034314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.66:40079 (local address 127.0.0.1:47922)
0929 03:21:28.034467 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0929 03:21:28.034509 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 03:21:28.034576 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:21:28.035195 (+ 619us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:21:28.035196 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:21:28.035198 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:21:28.035255 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:21:28.035257 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:21:28.036286 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:21:28.036287 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:21:28.037755 (+ 1468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:21:28.037757 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:21:28.039717 (+ 1960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:21:28.039718 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:21:28.039843 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:21:28.039849 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 03:21:28.039912 (+ 63us) client_negotiation.cc:240] Negotiation successful
0929 03:21:28.039959 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"mutex_wait_us":3,"thread_start_us":57,"threads_started":1}
I20260929 03:21:28.040337 8864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:28.034764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47922 (local address 127.20.178.66:40079)
0929 03:21:28.034969 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0929 03:21:28.034973 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:21:28.034983 (+ 10us) server_negotiation.cc:407] Connection header received
0929 03:21:28.035024 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:21:28.035026 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:21:28.035041 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:21:28.035120 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:21:28.035344 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:28.036184 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:28.038735 (+ 2551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:28.039630 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:28.039699 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:21:28.039790 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0929 03:21:28.040210 (+ 420us) server_negotiation.cc:299] Negotiation successful
0929 03:21:28.040255 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":108,"threads_started":1}
I20260929 03:21:28.040253 8785 tablet_service.cc:1467] Processing CreateTablet for tablet 4b02b1c2ffd1414a9e1fee0f037551b3 (DEFAULT_TABLE table=test-table [id=88306341fa1743a683df522de23d249e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:21:28.040585 8785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b02b1c2ffd1414a9e1fee0f037551b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:21:28.041298 8654 tablet_service.cc:1467] Processing CreateTablet for tablet 4b02b1c2ffd1414a9e1fee0f037551b3 (DEFAULT_TABLE table=test-table [id=88306341fa1743a683df522de23d249e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:21:28.041628 8654 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b02b1c2ffd1414a9e1fee0f037551b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:21:28.042996 8865 tablet_bootstrap.cc:492] T 4b02b1c2ffd1414a9e1fee0f037551b3 P 94305630600f48dbb68c3af48fd41bc4: Bootstrap starting.
I20260929 03:21:28.043582 8865 tablet_bootstrap.cc:654] T 4b02b1c2ffd1414a9e1fee0f037551b3 P 94305630600f48dbb68c3af48fd41bc4: Neither blocks nor log segments found. Creating new log.
I20260929 03:21:28.043871 8865 log.cc:824] T 4b02b1c2ffd1414a9e1fee0f037551b3 P 94305630600f48dbb68c3af48fd41bc4: Log is configured to *not* fsync() on all Append() calls
I20260929 03:21:28.044003 8866 tablet_bootstrap.cc:492] T 4b02b1c2ffd1414a9e1fee0f037551b3 P d311e7150f1b468da7512081f03958e7: Bootstrap starting.
I20260929 03:21:28.044580 8866 tablet_bootstrap.cc:654] T 4b02b1c2ffd1414a9e1fee0f037551b3 P d311e7150f1b468da7512081f03958e7: Neither blocks nor log segments found. Creating new log.
I20260929 03:21:28.044909 8868 tablet_bootstrap.cc:492] T 4b02b1c2ffd1414a9e1fee0f037551b3 P d610a77bd83c43bf9f11721fb5990a9d: Bootstrap starting.
I20260929 03:21:28.044956 8866 log.cc:824] T 4b02b1c2ffd1414a9e1fee0f037551b3 P d311e7150f1b468da7512081f03958e7: Log is configured to *not* fsync() on all Append() calls
I20260929 03:21:28.045572 8868 tablet_bootstrap.cc:654] T 4b02b1c2ffd1414a9e1fee0f037551b3 P d610a77bd83c43bf9f11721fb5990a9d: Neither blocks nor log segments found. Creating new log.
I20260929 03:21:28.045694 8866 tablet_bootstrap.cc:492] T 4b02b1c2ffd1414a9e1fee0f037551b3 P d311e7150f1b468da7512081f03958e7: No bootstrap required, opened a new log
I20260929 03:21:28.045753 8865 tablet_bootstrap.cc:492] T 4b02b1c2ffd1414a9e1fee0f037551b3 P 94305630600f48dbb68c3af48fd41bc4: No bootstrap required, opened a new log
I20260929 03:21:28.045773 8866 ts_tablet_manager.cc:1397] T 4b02b1c2ffd1414a9e1fee0f037551b3 P d311e7150f1b468da7512081f03958e7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 03:21:28.045802 8865 ts_tablet_manager.cc:1397] T 4b02b1c2ffd1414a9e1fee0f037551b3 P 94305630600f48dbb68c3af48fd41bc4: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260929 03:21:28.045957 8868 log.cc:824] T 4b02b1c2ffd1414a9e1fee0f037551b3 P d610a77bd83c43bf9f11721fb5990a9d: Log is configured to *not* fsync() on all Append() calls
I20260929 03:21:28.046721 8868 tablet_bootstrap.cc:492] T 4b02b1c2ffd1414a9e1fee0f037551b3 P d610a77bd83c43bf9f11721fb5990a9d: No bootstrap required, opened a new log
I20260929 03:21:28.046808 8868 ts_tablet_manager.cc:1397] T 4b02b1c2ffd1414a9e1fee0f037551b3 P d610a77bd83c43bf9f11721fb5990a9d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 03:21:28.048417 8865 raft_consensus.cc:348] T 4b02b1c2ffd1414a9e1fee0f037551b3 P 94305630600f48dbb68c3af48fd41bc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94305630600f48dbb68c3af48fd41bc4" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 44255 } } peers { permanent_uuid: "d610a77bd83c43bf9f11721fb5990a9d" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 40079 } } peers { permanent_uuid: "d311e7150f1b468da7512081f03958e7" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 45555 } }
I20260929 03:21:28.048425 8866 raft_consensus.cc:348] T 4b02b1c2ffd1414a9e1fee0f037551b3 P d311e7150f1b468da7512081f03958e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94305630600f48dbb68c3af48fd41bc4" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 44255 } } peers { permanent_uuid: "d610a77bd83c43bf9f11721fb5990a9d" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 40079 } } peers { permanent_uuid: "d311e7150f1b468da7512081f03958e7" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 45555 } }
I20260929 03:21:28.048622 8865 raft_consensus.cc:374] T 4b02b1c2ffd1414a9e1fee0f037551b3 P 94305630600f48dbb68c3af48fd41bc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:21:28.048623 8866 raft_consensus.cc:374] T 4b02b1c2ffd1414a9e1fee0f037551b3 P d311e7150f1b468da7512081f03958e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:21:28.048697 8866 raft_consensus.cc:729] T 4b02b1c2ffd1414a9e1fee0f037551b3 P d311e7150f1b468da7512081f03958e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d311e7150f1b468da7512081f03958e7, State: Initialized, Role: FOLLOWER
I20260929 03:21:28.048697 8865 raft_consensus.cc:729] T 4b02b1c2ffd1414a9e1fee0f037551b3 P 94305630600f48dbb68c3af48fd41bc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94305630600f48dbb68c3af48fd41bc4, State: Initialized, Role: FOLLOWER
I20260929 03:21:28.048804 8865 consensus_queue.cc:260] T 4b02b1c2ffd1414a9e1fee0f037551b3 P 94305630600f48dbb68c3af48fd41bc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94305630600f48dbb68c3af48fd41bc4" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 44255 } } peers { permanent_uuid: "d610a77bd83c43bf9f11721fb5990a9d" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 40079 } } peers { permanent_uuid: "d311e7150f1b468da7512081f03958e7" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 45555 } }
I20260929 03:21:28.048861 8866 consensus_queue.cc:260] T 4b02b1c2ffd1414a9e1fee0f037551b3 P d311e7150f1b468da7512081f03958e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94305630600f48dbb68c3af48fd41bc4" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 44255 } } peers { permanent_uuid: "d610a77bd83c43bf9f11721fb5990a9d" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 40079 } } peers { permanent_uuid: "d311e7150f1b468da7512081f03958e7" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 45555 } }
I20260929 03:21:28.049098 8866 ts_tablet_manager.cc:1428] T 4b02b1c2ffd1414a9e1fee0f037551b3 P d311e7150f1b468da7512081f03958e7: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20260929 03:21:28.049155 8851 heartbeater.cc:503] Master 127.20.178.126:33261 was elected leader, sending a full tablet report...
I20260929 03:21:28.049098 8865 ts_tablet_manager.cc:1428] T 4b02b1c2ffd1414a9e1fee0f037551b3 P 94305630600f48dbb68c3af48fd41bc4: Time spent starting tablet: real 0.003s user 0.000s sys 0.004s
I20260929 03:21:28.049597 8589 heartbeater.cc:503] Master 127.20.178.126:33261 was elected leader, sending a full tablet report...
I20260929 03:21:28.049527 8868 raft_consensus.cc:348] T 4b02b1c2ffd1414a9e1fee0f037551b3 P d610a77bd83c43bf9f11721fb5990a9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94305630600f48dbb68c3af48fd41bc4" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 44255 } } peers { permanent_uuid: "d610a77bd83c43bf9f11721fb5990a9d" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 40079 } } peers { permanent_uuid: "d311e7150f1b468da7512081f03958e7" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 45555 } }
I20260929 03:21:28.049710 8868 raft_consensus.cc:374] T 4b02b1c2ffd1414a9e1fee0f037551b3 P d610a77bd83c43bf9f11721fb5990a9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:21:28.049757 8868 raft_consensus.cc:729] T 4b02b1c2ffd1414a9e1fee0f037551b3 P d610a77bd83c43bf9f11721fb5990a9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d610a77bd83c43bf9f11721fb5990a9d, State: Initialized, Role: FOLLOWER
I20260929 03:21:28.049873 8868 consensus_queue.cc:260] T 4b02b1c2ffd1414a9e1fee0f037551b3 P d610a77bd83c43bf9f11721fb5990a9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94305630600f48dbb68c3af48fd41bc4" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 44255 } } peers { permanent_uuid: "d610a77bd83c43bf9f11721fb5990a9d" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 40079 } } peers { permanent_uuid: "d311e7150f1b468da7512081f03958e7" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 45555 } }
I20260929 03:21:28.050139 8868 ts_tablet_manager.cc:1428] T 4b02b1c2ffd1414a9e1fee0f037551b3 P d610a77bd83c43bf9f11721fb5990a9d: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260929 03:21:28.050359 8720 heartbeater.cc:503] Master 127.20.178.126:33261 was elected leader, sending a full tablet report...
I20260929 03:21:28.128221 8871 raft_consensus.cc:482] T 4b02b1c2ffd1414a9e1fee0f037551b3 P 94305630600f48dbb68c3af48fd41bc4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:21:28.128394 8871 raft_consensus.cc:504] T 4b02b1c2ffd1414a9e1fee0f037551b3 P 94305630600f48dbb68c3af48fd41bc4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94305630600f48dbb68c3af48fd41bc4" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 44255 } } peers { permanent_uuid: "d610a77bd83c43bf9f11721fb5990a9d" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 40079 } } peers { permanent_uuid: "d311e7150f1b468da7512081f03958e7" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 45555 } }
I20260929 03:21:28.128846 8871 leader_election.cc:288] T 4b02b1c2ffd1414a9e1fee0f037551b3 P 94305630600f48dbb68c3af48fd41bc4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d610a77bd83c43bf9f11721fb5990a9d (127.20.178.66:40079), d311e7150f1b468da7512081f03958e7 (127.20.178.67:45555)
I20260929 03:21:28.131565 8864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:28.129124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.65:37865 (local address 127.20.178.66:40079)
0929 03:21:28.129174 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0929 03:21:28.129177 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:21:28.129187 (+ 10us) server_negotiation.cc:407] Connection header received
0929 03:21:28.129217 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:21:28.129218 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:21:28.129228 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:21:28.129286 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:21:28.129451 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:28.129924 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:28.130850 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:28.131212 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:28.131307 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:21:28.131315 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0929 03:21:28.131460 (+ 145us) server_negotiation.cc:299] Negotiation successful
0929 03:21:28.131495 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260929 03:21:28.131577 8591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:28.129011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.66:40079 (local address 127.20.178.65:37865)
0929 03:21:28.129077 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0929 03:21:28.129091 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 03:21:28.129136 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:21:28.129290 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:21:28.129291 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:21:28.129293 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:21:28.129377 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:21:28.129378 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:21:28.130009 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:21:28.130010 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:21:28.130750 (+ 740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:21:28.130752 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:21:28.131307 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:21:28.131308 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:21:28.131385 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:21:28.131388 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 03:21:28.131454 (+ 66us) client_negotiation.cc:240] Negotiation successful
0929 03:21:28.131515 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40,"mutex_wait_us":17}
I20260929 03:21:28.131747 8874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:28.129136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.67:45555 (local address 127.20.178.65:46453)
0929 03:21:28.129294 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0929 03:21:28.129303 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 03:21:28.129358 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:21:28.129473 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:21:28.129473 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:21:28.129475 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:21:28.129543 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:21:28.129544 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:21:28.130160 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:21:28.130161 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:21:28.130906 (+ 745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:21:28.130908 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:21:28.131438 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:21:28.131438 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:21:28.131523 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:21:28.131528 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 03:21:28.131595 (+ 67us) client_negotiation.cc:240] Negotiation successful
0929 03:21:28.131635 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":98,"threads_started":1}
I20260929 03:21:28.131914 8863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:28.129208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.178.65:46453 (local address 127.20.178.67:45555)
0929 03:21:28.129253 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0929 03:21:28.129255 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:21:28.129367 (+ 112us) server_negotiation.cc:407] Connection header received
0929 03:21:28.129411 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:21:28.129412 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:21:28.129421 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:21:28.129471 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:21:28.129622 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:28.130079 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:28.131000 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:28.131357 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:28.131437 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:21:28.131444 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0929 03:21:28.131736 (+ 292us) server_negotiation.cc:299] Negotiation successful
0929 03:21:28.131793 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260929 03:21:28.131918 8674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b02b1c2ffd1414a9e1fee0f037551b3" candidate_uuid: "94305630600f48dbb68c3af48fd41bc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d610a77bd83c43bf9f11721fb5990a9d" is_pre_election: true
I20260929 03:21:28.132072 8674 raft_consensus.cc:2454] T 4b02b1c2ffd1414a9e1fee0f037551b3 P d610a77bd83c43bf9f11721fb5990a9d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 94305630600f48dbb68c3af48fd41bc4 in term 0.
I20260929 03:21:28.132337 8479 leader_election.cc:302] T 4b02b1c2ffd1414a9e1fee0f037551b3 P 94305630600f48dbb68c3af48fd41bc4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 94305630600f48dbb68c3af48fd41bc4, d610a77bd83c43bf9f11721fb5990a9d; no voters:
I20260929 03:21:28.132441 8805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b02b1c2ffd1414a9e1fee0f037551b3" candidate_uuid: "94305630600f48dbb68c3af48fd41bc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d311e7150f1b468da7512081f03958e7" is_pre_election: true
I20260929 03:21:28.132494 8871 raft_consensus.cc:2781] T 4b02b1c2ffd1414a9e1fee0f037551b3 P 94305630600f48dbb68c3af48fd41bc4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 03:21:28.132537 8871 raft_consensus.cc:482] T 4b02b1c2ffd1414a9e1fee0f037551b3 P 94305630600f48dbb68c3af48fd41bc4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 03:21:28.132558 8871 raft_consensus.cc:3037] T 4b02b1c2ffd1414a9e1fee0f037551b3 P 94305630600f48dbb68c3af48fd41bc4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:21:28.132589 8805 raft_consensus.cc:2454] T 4b02b1c2ffd1414a9e1fee0f037551b3 P d311e7150f1b468da7512081f03958e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 94305630600f48dbb68c3af48fd41bc4 in term 0.
I20260929 03:21:28.133338 8871 raft_consensus.cc:504] T 4b02b1c2ffd1414a9e1fee0f037551b3 P 94305630600f48dbb68c3af48fd41bc4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94305630600f48dbb68c3af48fd41bc4" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 44255 } } peers { permanent_uuid: "d610a77bd83c43bf9f11721fb5990a9d" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 40079 } } peers { permanent_uuid: "d311e7150f1b468da7512081f03958e7" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 45555 } }
I20260929 03:21:28.133447 8871 leader_election.cc:288] T 4b02b1c2ffd1414a9e1fee0f037551b3 P 94305630600f48dbb68c3af48fd41bc4 [CANDIDATE]: Term 1 election: Requested vote from peers d610a77bd83c43bf9f11721fb5990a9d (127.20.178.66:40079), d311e7150f1b468da7512081f03958e7 (127.20.178.67:45555)
I20260929 03:21:28.133665 8674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b02b1c2ffd1414a9e1fee0f037551b3" candidate_uuid: "94305630600f48dbb68c3af48fd41bc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d610a77bd83c43bf9f11721fb5990a9d"
I20260929 03:21:28.133699 8805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b02b1c2ffd1414a9e1fee0f037551b3" candidate_uuid: "94305630600f48dbb68c3af48fd41bc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d311e7150f1b468da7512081f03958e7"
I20260929 03:21:28.133746 8674 raft_consensus.cc:3037] T 4b02b1c2ffd1414a9e1fee0f037551b3 P d610a77bd83c43bf9f11721fb5990a9d [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:21:28.133755 8805 raft_consensus.cc:3037] T 4b02b1c2ffd1414a9e1fee0f037551b3 P d311e7150f1b468da7512081f03958e7 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:21:28.134464 8805 raft_consensus.cc:2454] T 4b02b1c2ffd1414a9e1fee0f037551b3 P d311e7150f1b468da7512081f03958e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94305630600f48dbb68c3af48fd41bc4 in term 1.
I20260929 03:21:28.134476 8674 raft_consensus.cc:2454] T 4b02b1c2ffd1414a9e1fee0f037551b3 P d610a77bd83c43bf9f11721fb5990a9d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94305630600f48dbb68c3af48fd41bc4 in term 1.
I20260929 03:21:28.134617 8479 leader_election.cc:302] T 4b02b1c2ffd1414a9e1fee0f037551b3 P 94305630600f48dbb68c3af48fd41bc4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 94305630600f48dbb68c3af48fd41bc4, d610a77bd83c43bf9f11721fb5990a9d; no voters:
I20260929 03:21:28.134747 8871 raft_consensus.cc:2781] T 4b02b1c2ffd1414a9e1fee0f037551b3 P 94305630600f48dbb68c3af48fd41bc4 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:21:28.134899 8871 raft_consensus.cc:686] T 4b02b1c2ffd1414a9e1fee0f037551b3 P 94305630600f48dbb68c3af48fd41bc4 [term 1 LEADER]: Becoming Leader. State: Replica: 94305630600f48dbb68c3af48fd41bc4, State: Running, Role: LEADER
I20260929 03:21:28.135118 8871 consensus_queue.cc:237] T 4b02b1c2ffd1414a9e1fee0f037551b3 P 94305630600f48dbb68c3af48fd41bc4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94305630600f48dbb68c3af48fd41bc4" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 44255 } } peers { permanent_uuid: "d610a77bd83c43bf9f11721fb5990a9d" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 40079 } } peers { permanent_uuid: "d311e7150f1b468da7512081f03958e7" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 45555 } }
W20260929 03:21:28.135342 8721 tablet.cc:2369] T 4b02b1c2ffd1414a9e1fee0f037551b3 P d610a77bd83c43bf9f11721fb5990a9d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 03:21:28.136025 8401 catalog_manager.cc:5495] T 4b02b1c2ffd1414a9e1fee0f037551b3 P 94305630600f48dbb68c3af48fd41bc4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 94305630600f48dbb68c3af48fd41bc4 (127.20.178.65). New cstate: current_term: 1 leader_uuid: "94305630600f48dbb68c3af48fd41bc4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94305630600f48dbb68c3af48fd41bc4" member_type: VOTER last_known_addr { host: "127.20.178.65" port: 44255 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d610a77bd83c43bf9f11721fb5990a9d" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 40079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d311e7150f1b468da7512081f03958e7" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 45555 } health_report { overall_health: UNKNOWN } } }
I20260929 03:21:28.155865 8861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:28.153702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34666 (local address 127.20.178.65:44255)
0929 03:21:28.153756 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0929 03:21:28.153760 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:21:28.153772 (+ 12us) server_negotiation.cc:407] Connection header received
0929 03:21:28.153804 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:21:28.153804 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:21:28.153824 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:21:28.153910 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:21:28.154031 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:28.154506 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:28.155219 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:28.155406 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:28.155457 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:21:28.155583 (+ 126us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:21:28.155678 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:21:28.155737 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0929 03:21:28.155773 (+ 36us) server_negotiation.cc:299] Negotiation successful
0929 03:21:28.155793 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260929 03:21:28.157655 8674 raft_consensus.cc:1261] T 4b02b1c2ffd1414a9e1fee0f037551b3 P d610a77bd83c43bf9f11721fb5990a9d [term 1 FOLLOWER]: Refusing update from remote peer 94305630600f48dbb68c3af48fd41bc4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:21:28.157673 8805 raft_consensus.cc:1261] T 4b02b1c2ffd1414a9e1fee0f037551b3 P d311e7150f1b468da7512081f03958e7 [term 1 FOLLOWER]: Refusing update from remote peer 94305630600f48dbb68c3af48fd41bc4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:21:28.158041 8871 consensus_queue.cc:1035] T 4b02b1c2ffd1414a9e1fee0f037551b3 P 94305630600f48dbb68c3af48fd41bc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d610a77bd83c43bf9f11721fb5990a9d" member_type: VOTER last_known_addr { host: "127.20.178.66" port: 40079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:21:28.158191 8875 consensus_queue.cc:1035] T 4b02b1c2ffd1414a9e1fee0f037551b3 P 94305630600f48dbb68c3af48fd41bc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d311e7150f1b468da7512081f03958e7" member_type: VOTER last_known_addr { host: "127.20.178.67" port: 45555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:21:28.160611 8879 mvcc.cc:205] Tried to move back new op lower bound from 7334510953091051520 to 7334510953002057728. Current Snapshot: MvccSnapshot[applied={T|T < 7334510953091051520}]
I20260929 03:21:28.160797 8880 mvcc.cc:205] Tried to move back new op lower bound from 7334510953091051520 to 7334510953002057728. Current Snapshot: MvccSnapshot[applied={T|T < 7334510953091051520}]
I20260929 03:21:28.163290 8401 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:48912:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:21:28.163398 8401 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:48912:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:21:28.164181 8401 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fec78c577ee147fba89b0a315218406a: Sending DeleteTablet for 3 replicas of tablet 4b02b1c2ffd1414a9e1fee0f037551b3
I20260929 03:21:28.164548 8785 tablet_service.cc:1514] Processing DeleteTablet for tablet 4b02b1c2ffd1414a9e1fee0f037551b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:21:28 UTC) from {username='slave'} at 127.0.0.1:54666
I20260929 03:21:28.164577 8523 tablet_service.cc:1514] Processing DeleteTablet for tablet 4b02b1c2ffd1414a9e1fee0f037551b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:21:28 UTC) from {username='slave'} at 127.0.0.1:34654
I20260929 03:21:28.164579 8654 tablet_service.cc:1514] Processing DeleteTablet for tablet 4b02b1c2ffd1414a9e1fee0f037551b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:21:28 UTC) from {username='slave'} at 127.0.0.1:47922
I20260929 03:21:28.164808 8889 tablet_replica.cc:330] T 4b02b1c2ffd1414a9e1fee0f037551b3 P 94305630600f48dbb68c3af48fd41bc4: stopping tablet replica
I20260929 03:21:28.164911 8889 raft_consensus.cc:2229] T 4b02b1c2ffd1414a9e1fee0f037551b3 P 94305630600f48dbb68c3af48fd41bc4 [term 1 LEADER]: Raft consensus shutting down.
I20260929 03:21:28.165032 8889 raft_consensus.cc:2258] T 4b02b1c2ffd1414a9e1fee0f037551b3 P 94305630600f48dbb68c3af48fd41bc4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:21:28.165112 21193 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuoaHbX/build/release/bin/kudu with pid 8461
I20260929 03:21:28.165867 8890 tablet_replica.cc:330] T 4b02b1c2ffd1414a9e1fee0f037551b3 P d610a77bd83c43bf9f11721fb5990a9d: stopping tablet replica
I20260929 03:21:28.165992 8890 raft_consensus.cc:2229] T 4b02b1c2ffd1414a9e1fee0f037551b3 P d610a77bd83c43bf9f11721fb5990a9d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:21:28.166095 8890 raft_consensus.cc:2258] T 4b02b1c2ffd1414a9e1fee0f037551b3 P d610a77bd83c43bf9f11721fb5990a9d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:21:28.166499 8890 ts_tablet_manager.cc:1905] T 4b02b1c2ffd1414a9e1fee0f037551b3 P d610a77bd83c43bf9f11721fb5990a9d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:21:28.168098 8890 ts_tablet_manager.cc:1918] T 4b02b1c2ffd1414a9e1fee0f037551b3 P d610a77bd83c43bf9f11721fb5990a9d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:21:28.168185 8890 log.cc:1196] T 4b02b1c2ffd1414a9e1fee0f037551b3 P d610a77bd83c43bf9f11721fb5990a9d: Deleting WAL directory at /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790651852476208-21193-0/minicluster-data/ts-1/wal/wals/4b02b1c2ffd1414a9e1fee0f037551b3
I20260929 03:21:28.168416 8890 ts_tablet_manager.cc:1939] T 4b02b1c2ffd1414a9e1fee0f037551b3 P d610a77bd83c43bf9f11721fb5990a9d: Deleting consensus metadata
I20260929 03:21:28.168879 8388 catalog_manager.cc:4841] TS d610a77bd83c43bf9f11721fb5990a9d (127.20.178.66:40079): tablet 4b02b1c2ffd1414a9e1fee0f037551b3 (table test-table [id=88306341fa1743a683df522de23d249e]) successfully deleted
W20260929 03:21:28.170929 8386 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.20.178.65:44255 (error 108)
I20260929 03:21:28.171157 21193 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuoaHbX/build/release/bin/kudu with pid 8592
I20260929 03:21:28.171164 8888 tablet_replica.cc:330] T 4b02b1c2ffd1414a9e1fee0f037551b3 P d311e7150f1b468da7512081f03958e7: stopping tablet replica
I20260929 03:21:28.171734 8859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:28.171256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.65:44255 (local address 127.0.0.1:34678)
0929 03:21:28.171653 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0929 03:21:28.171693 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.178.65:44255: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":375}
I20260929 03:21:28.171752 8888 raft_consensus.cc:2229] T 4b02b1c2ffd1414a9e1fee0f037551b3 P d311e7150f1b468da7512081f03958e7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:21:28.172037 8888 raft_consensus.cc:2258] T 4b02b1c2ffd1414a9e1fee0f037551b3 P d311e7150f1b468da7512081f03958e7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:21:28.172871 8888 ts_tablet_manager.cc:1905] T 4b02b1c2ffd1414a9e1fee0f037551b3 P d311e7150f1b468da7512081f03958e7: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260929 03:21:28.173417 8386 catalog_manager.cc:4584] TS 94305630600f48dbb68c3af48fd41bc4 (127.20.178.65:44255): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4b02b1c2ffd1414a9e1fee0f037551b3: Network error: Client connection negotiation failed: client connection to 127.20.178.65:44255: connect: Connection refused (error 111)
I20260929 03:21:28.174844 8888 ts_tablet_manager.cc:1918] T 4b02b1c2ffd1414a9e1fee0f037551b3 P d311e7150f1b468da7512081f03958e7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:21:28.174923 8888 log.cc:1196] T 4b02b1c2ffd1414a9e1fee0f037551b3 P d311e7150f1b468da7512081f03958e7: Deleting WAL directory at /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790651852476208-21193-0/minicluster-data/ts-2/wal/wals/4b02b1c2ffd1414a9e1fee0f037551b3
I20260929 03:21:28.175175 8888 ts_tablet_manager.cc:1939] T 4b02b1c2ffd1414a9e1fee0f037551b3 P d311e7150f1b468da7512081f03958e7: Deleting consensus metadata
I20260929 03:21:28.175583 8388 catalog_manager.cc:4841] TS d311e7150f1b468da7512081f03958e7 (127.20.178.67:45555): tablet 4b02b1c2ffd1414a9e1fee0f037551b3 (table test-table [id=88306341fa1743a683df522de23d249e]) successfully deleted
I20260929 03:21:28.176612 21193 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuoaHbX/build/release/bin/kudu with pid 8723
I20260929 03:21:28.183260 21193 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuoaHbX/build/release/bin/kudu with pid 8370
2026-09-29T03:21:28Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (754 ms)
[----------] 6 tests from EncryptionPolicyTest (7909 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790651852476208-21193-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:21:28 dist-test-slave-c9qh krb5kdc[8894](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:21:28 dist-test-slave-c9qh krb5kdc[8894](info): set up 2 sockets
Sep 29 03:21:28 dist-test-slave-c9qh krb5kdc[8894](info): commencing operation
krb5kdc: starting...
W20260929 03:21:29.216725 21193 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790651852476208-21193-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790651852476208-21193-0/krb5kdc/test-user.keytab.
Sep 29 03:21:29 dist-test-slave-c9qh krb5kdc[8894](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652089, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:21:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:21:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790651852476208-21193-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790651852476208-21193-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 03:21:29.332984 21193 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuoaHbX/build/release/bin/kudu
/tmp/dist-test-taskuoaHbX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790651852476208-21193-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790651852476208-21193-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790651852476208-21193-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790651852476208-21193-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36553
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790651852476208-21193-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36553
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790651852476208-21193-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790651852476208-21193-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790651852476208-21193-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:21:29.417424 8907 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:21:29.417646 8907 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:21:29.417683 8907 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:21:29.418840 8907 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:21:29.418907 8907 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:21:29.418933 8907 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:21:29.418964 8907 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:21:29.418991 8907 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:21:29.420477 8907 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790651852476208-21193-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790651852476208-21193-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36553
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790651852476208-21193-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36553
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790651852476208-21193-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8907
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790651852476208-21193-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:21:29.420765 8907 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:21:29.420993 8907 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:21:29.425668 8912 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:21:29.425679 8913 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:21:29.425732 8915 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:21:29.426036 8907 server_base.cc:1034] running on GCE node
I20260929 03:21:29.426221 8907 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:21:29.426515 8907 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:21:29.427654 8907 hybrid_clock.cc:648] HybridClock initialized: now 1790652089427641 us; error 25 us; skew 500 ppm
Sep 29 03:21:29 dist-test-slave-c9qh krb5kdc[8894](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652089, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:21:29.430070 8907 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 03:21:29.430491 8907 webserver.cc:460] Webserver started at http://127.0.0.1:37219/ using document root <none> and password file <none>
I20260929 03:21:29.430665 8907 fs_manager.cc:359] Metadata directory not provided
I20260929 03:21:29.430702 8907 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:21:29.430785 8907 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:21:29.431723 8907 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790651852476208-21193-0/minicluster-data/master-0/data/instance:
uuid: "fcd1f71caeb9423e82d0b9650ff9485d"
format_stamp: "Formatted at 2026-09-29 03:21:29 on dist-test-slave-c9qh"
server_key: "b7d771c15a85c09979cf4cd2cf2639f7"
server_key_iv: "420fd6c5e81030c5bbcb3ad1b3445b04"
server_key_version: "encryptionkey@0"
I20260929 03:21:29.432106 8907 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790651852476208-21193-0/minicluster-data/master-0/wal/instance:
uuid: "fcd1f71caeb9423e82d0b9650ff9485d"
format_stamp: "Formatted at 2026-09-29 03:21:29 on dist-test-slave-c9qh"
server_key: "b7d771c15a85c09979cf4cd2cf2639f7"
server_key_iv: "420fd6c5e81030c5bbcb3ad1b3445b04"
server_key_version: "encryptionkey@0"
I20260929 03:21:29.433787 8907 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 03:21:29.434502 8922 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:21:29.434674 8907 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 03:21:29.434775 8907 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790651852476208-21193-0/minicluster-data/master-0/data,/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790651852476208-21193-0/minicluster-data/master-0/wal
uuid: "fcd1f71caeb9423e82d0b9650ff9485d"
format_stamp: "Formatted at 2026-09-29 03:21:29 on dist-test-slave-c9qh"
server_key: "b7d771c15a85c09979cf4cd2cf2639f7"
server_key_iv: "420fd6c5e81030c5bbcb3ad1b3445b04"
server_key_version: "encryptionkey@0"
I20260929 03:21:29.434867 8907 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790651852476208-21193-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790651852476208-21193-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790651852476208-21193-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:21:29.447875 8907 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:21:29.449751 8907 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:21:29.449918 8907 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:21:29.454355 8907 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36553
I20260929 03:21:29.454386 8974 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36553 every 8 connection(s)
I20260929 03:21:29.454737 8907 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790651852476208-21193-0/minicluster-data/master-0/data/info.pb
I20260929 03:21:29.455592 8975 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:21:29.457634 21193 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuoaHbX/build/release/bin/kudu as pid 8907
I20260929 03:21:29.457736 21193 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790651852476208-21193-0/minicluster-data/master-0/wal/instance
I20260929 03:21:29.457935 21193 external_mini_cluster.cc:1428] Setting key 9dfd5beb70afeab353e566f8e50c13dd
I20260929 03:21:29.458154 8975 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fcd1f71caeb9423e82d0b9650ff9485d: Bootstrap starting.
I20260929 03:21:29.458837 8975 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fcd1f71caeb9423e82d0b9650ff9485d: Neither blocks nor log segments found. Creating new log.
I20260929 03:21:29.459141 8975 log.cc:824] T 00000000000000000000000000000000 P fcd1f71caeb9423e82d0b9650ff9485d: Log is configured to *not* fsync() on all Append() calls
I20260929 03:21:29.459777 8975 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fcd1f71caeb9423e82d0b9650ff9485d: No bootstrap required, opened a new log
I20260929 03:21:29.461617 8975 raft_consensus.cc:348] T 00000000000000000000000000000000 P fcd1f71caeb9423e82d0b9650ff9485d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcd1f71caeb9423e82d0b9650ff9485d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36553 } }
I20260929 03:21:29.461746 8975 raft_consensus.cc:374] T 00000000000000000000000000000000 P fcd1f71caeb9423e82d0b9650ff9485d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:21:29.461767 8975 raft_consensus.cc:729] T 00000000000000000000000000000000 P fcd1f71caeb9423e82d0b9650ff9485d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fcd1f71caeb9423e82d0b9650ff9485d, State: Initialized, Role: FOLLOWER
I20260929 03:21:29.461894 8975 consensus_queue.cc:260] T 00000000000000000000000000000000 P fcd1f71caeb9423e82d0b9650ff9485d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcd1f71caeb9423e82d0b9650ff9485d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36553 } }
I20260929 03:21:29.461964 8975 raft_consensus.cc:388] T 00000000000000000000000000000000 P fcd1f71caeb9423e82d0b9650ff9485d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:21:29.461992 8975 raft_consensus.cc:482] T 00000000000000000000000000000000 P fcd1f71caeb9423e82d0b9650ff9485d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:21:29.462042 8975 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fcd1f71caeb9423e82d0b9650ff9485d [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:21:29.462587 8975 raft_consensus.cc:504] T 00000000000000000000000000000000 P fcd1f71caeb9423e82d0b9650ff9485d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcd1f71caeb9423e82d0b9650ff9485d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36553 } }
I20260929 03:21:29.462708 8975 leader_election.cc:302] T 00000000000000000000000000000000 P fcd1f71caeb9423e82d0b9650ff9485d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fcd1f71caeb9423e82d0b9650ff9485d; no voters:
I20260929 03:21:29.462893 8975 leader_election.cc:288] T 00000000000000000000000000000000 P fcd1f71caeb9423e82d0b9650ff9485d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:21:29.462919 8980 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fcd1f71caeb9423e82d0b9650ff9485d [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:21:29.463028 8980 raft_consensus.cc:686] T 00000000000000000000000000000000 P fcd1f71caeb9423e82d0b9650ff9485d [term 1 LEADER]: Becoming Leader. State: Replica: fcd1f71caeb9423e82d0b9650ff9485d, State: Running, Role: LEADER
I20260929 03:21:29.463105 8980 consensus_queue.cc:237] T 00000000000000000000000000000000 P fcd1f71caeb9423e82d0b9650ff9485d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcd1f71caeb9423e82d0b9650ff9485d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36553 } }
I20260929 03:21:29.463155 8975 sys_catalog.cc:564] T 00000000000000000000000000000000 P fcd1f71caeb9423e82d0b9650ff9485d [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:21:29.463420 8980 sys_catalog.cc:455] T 00000000000000000000000000000000 P fcd1f71caeb9423e82d0b9650ff9485d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fcd1f71caeb9423e82d0b9650ff9485d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcd1f71caeb9423e82d0b9650ff9485d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36553 } } }
I20260929 03:21:29.463478 8980 sys_catalog.cc:458] T 00000000000000000000000000000000 P fcd1f71caeb9423e82d0b9650ff9485d [sys.catalog]: This master's current role is: LEADER
I20260929 03:21:29.463599 8980 sys_catalog.cc:455] T 00000000000000000000000000000000 P fcd1f71caeb9423e82d0b9650ff9485d [sys.catalog]: SysCatalogTable state changed. Reason: New leader fcd1f71caeb9423e82d0b9650ff9485d. Latest consensus state: current_term: 1 leader_uuid: "fcd1f71caeb9423e82d0b9650ff9485d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcd1f71caeb9423e82d0b9650ff9485d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36553 } } }
I20260929 03:21:29.463656 8980 sys_catalog.cc:458] T 00000000000000000000000000000000 P fcd1f71caeb9423e82d0b9650ff9485d [sys.catalog]: This master's current role is: LEADER
Sep 29 03:21:29 dist-test-slave-c9qh krb5kdc[8894](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652089, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
W20260929 03:21:29.464980 8995 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fcd1f71caeb9423e82d0b9650ff9485d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 03:21:29.465448 8995 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 03:21:29.465523 8984 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:21:29.465736 8984 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:21:29.469430 8978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:29.458473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53325 (local address 127.0.0.1:36553)
0929 03:21:29.458669 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0929 03:21:29.458678 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 03:21:29.458716 (+ 38us) server_negotiation.cc:407] Connection header received
0929 03:21:29.459174 (+ 458us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:21:29.459201 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:21:29.459269 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:21:29.459423 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:21:29.460243 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:29.461185 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:29.462104 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:29.462509 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:29.462630 (+ 121us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:21:29.465752 (+ 3122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:21:29.465756 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:21:29.465756 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:21:29.465766 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:21:29.467720 (+ 1954us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:21:29.468138 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:21:29.468142 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:21:29.468142 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:21:29.468246 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:21:29.468471 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:21:29.468472 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:21:29.468472 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:21:29.468821 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:21:29.468979 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0929 03:21:29.469209 (+ 230us) server_negotiation.cc:299] Negotiation successful
0929 03:21:29.469285 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":87,"threads_started":1}
I20260929 03:21:29.469436 8984 catalog_manager.cc:1347] Generated new cluster ID: e8ec5a608b25430db55c1186412c671e
I20260929 03:21:29.469521 8984 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:21:29.470463 21193 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 03:21:29.482841 8978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:29.472950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50277 (local address 127.0.0.1:36553)
0929 03:21:29.472991 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0929 03:21:29.472993 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:21:29.476802 (+ 3809us) server_negotiation.cc:407] Connection header received
0929 03:21:29.476862 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:21:29.476863 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:21:29.476873 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:21:29.476926 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:21:29.477437 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:29.477994 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:29.478714 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:29.478957 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:29.479034 (+ 77us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:21:29.480223 (+ 1189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:21:29.480224 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:21:29.480225 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:21:29.480232 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:21:29.481663 (+ 1431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:21:29.482038 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:21:29.482039 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:21:29.482039 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:21:29.482098 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:21:29.482318 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:21:29.482318 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:21:29.482319 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:21:29.482430 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:21:29.482556 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0929 03:21:29.482662 (+ 106us) server_negotiation.cc:299] Negotiation successful
0929 03:21:29.482724 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260929 03:21:29.484845 8984 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:21:29.485327 8984 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:21:29.489465 8984 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fcd1f71caeb9423e82d0b9650ff9485d: Generated new TSK 0
I20260929 03:21:29.489651 8984 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 03:21:29.496182 21193 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuoaHbX/build/release/bin/kudu with pid 8907
W20260929 03:21:29.500517 8892 connection.cc:535] client connection to 127.0.0.1:36553 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36553): Connection reset by peer (error 104)
2026-09-29T03:21:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1311 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790651852476208-21193-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:21:29 dist-test-slave-c9qh krb5kdc[9006](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:21:29 dist-test-slave-c9qh krb5kdc[9006](info): set up 2 sockets
Sep 29 03:21:29 dist-test-slave-c9qh krb5kdc[9006](info): commencing operation
krb5kdc: starting...
W20260929 03:21:30.532539 21193 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790651852476208-21193-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790651852476208-21193-0/krb5kdc/test-user.keytab.
Sep 29 03:21:30 dist-test-slave-c9qh krb5kdc[9006](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652090, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:21:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:21:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790651852476208-21193-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790651852476208-21193-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 03:21:30.647207 21193 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuoaHbX/build/release/bin/kudu
/tmp/dist-test-taskuoaHbX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790651852476208-21193-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790651852476208-21193-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790651852476208-21193-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790651852476208-21193-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34489
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790651852476208-21193-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34489
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790651852476208-21193-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790651852476208-21193-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790651852476208-21193-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:21:30.725519 9019 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:21:30.725709 9019 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:21:30.725739 9019 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:21:30.726851 9019 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:21:30.726889 9019 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:21:30.726902 9019 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:21:30.726936 9019 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:21:30.726965 9019 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:21:30.728410 9019 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790651852476208-21193-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790651852476208-21193-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34489
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790651852476208-21193-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34489
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790651852476208-21193-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9019
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790651852476208-21193-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:21:30.728652 9019 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:21:30.728861 9019 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:21:30.733554 9024 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:21:30.733608 9019 server_base.cc:1034] running on GCE node
W20260929 03:21:30.733744 9027 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:21:30.733775 9025 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:21:30.734010 9019 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:21:30.734277 9019 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:21:30.735437 9019 hybrid_clock.cc:648] HybridClock initialized: now 1790652090735427 us; error 31 us; skew 500 ppm
Sep 29 03:21:30 dist-test-slave-c9qh krb5kdc[9006](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652090, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:21:30.737944 9019 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 03:21:30.738341 9019 webserver.cc:460] Webserver started at http://127.0.0.1:45645/ using document root <none> and password file <none>
I20260929 03:21:30.738587 9019 fs_manager.cc:359] Metadata directory not provided
I20260929 03:21:30.738654 9019 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:21:30.738803 9019 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:21:30.739784 9019 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790651852476208-21193-0/minicluster-data/master-0/data/instance:
uuid: "a552c8c140064cee859bb1b2185cd92a"
format_stamp: "Formatted at 2026-09-29 03:21:30 on dist-test-slave-c9qh"
server_key: "7c86da6f8671d924d067a18a07d4fb09"
server_key_iv: "b6cdba06801d97f8a1288b154f20d264"
server_key_version: "encryptionkey@0"
I20260929 03:21:30.740134 9019 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790651852476208-21193-0/minicluster-data/master-0/wal/instance:
uuid: "a552c8c140064cee859bb1b2185cd92a"
format_stamp: "Formatted at 2026-09-29 03:21:30 on dist-test-slave-c9qh"
server_key: "7c86da6f8671d924d067a18a07d4fb09"
server_key_iv: "b6cdba06801d97f8a1288b154f20d264"
server_key_version: "encryptionkey@0"
I20260929 03:21:30.741608 9019 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 03:21:30.742370 9034 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:21:30.742558 9019 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 03:21:30.742657 9019 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790651852476208-21193-0/minicluster-data/master-0/data,/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790651852476208-21193-0/minicluster-data/master-0/wal
uuid: "a552c8c140064cee859bb1b2185cd92a"
format_stamp: "Formatted at 2026-09-29 03:21:30 on dist-test-slave-c9qh"
server_key: "7c86da6f8671d924d067a18a07d4fb09"
server_key_iv: "b6cdba06801d97f8a1288b154f20d264"
server_key_version: "encryptionkey@0"
I20260929 03:21:30.742734 9019 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790651852476208-21193-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790651852476208-21193-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790651852476208-21193-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:21:30.765199 9019 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:21:30.767102 9019 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:21:30.767264 9019 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:21:30.771102 9019 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34489
I20260929 03:21:30.771142 9086 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34489 every 8 connection(s)
I20260929 03:21:30.771553 9019 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790651852476208-21193-0/minicluster-data/master-0/data/info.pb
I20260929 03:21:30.772532 9087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:21:30.774976 9087 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a552c8c140064cee859bb1b2185cd92a: Bootstrap starting.
I20260929 03:21:30.775606 9087 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a552c8c140064cee859bb1b2185cd92a: Neither blocks nor log segments found. Creating new log.
I20260929 03:21:30.775902 9087 log.cc:824] T 00000000000000000000000000000000 P a552c8c140064cee859bb1b2185cd92a: Log is configured to *not* fsync() on all Append() calls
I20260929 03:21:30.776724 9087 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a552c8c140064cee859bb1b2185cd92a: No bootstrap required, opened a new log
I20260929 03:21:30.778455 9087 raft_consensus.cc:348] T 00000000000000000000000000000000 P a552c8c140064cee859bb1b2185cd92a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a552c8c140064cee859bb1b2185cd92a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34489 } }
I20260929 03:21:30.778597 9087 raft_consensus.cc:374] T 00000000000000000000000000000000 P a552c8c140064cee859bb1b2185cd92a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:21:30.778621 9087 raft_consensus.cc:729] T 00000000000000000000000000000000 P a552c8c140064cee859bb1b2185cd92a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a552c8c140064cee859bb1b2185cd92a, State: Initialized, Role: FOLLOWER
I20260929 03:21:30.778762 9087 consensus_queue.cc:260] T 00000000000000000000000000000000 P a552c8c140064cee859bb1b2185cd92a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a552c8c140064cee859bb1b2185cd92a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34489 } }
I20260929 03:21:30.778847 9087 raft_consensus.cc:388] T 00000000000000000000000000000000 P a552c8c140064cee859bb1b2185cd92a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:21:30.778872 9087 raft_consensus.cc:482] T 00000000000000000000000000000000 P a552c8c140064cee859bb1b2185cd92a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:21:30.778903 9087 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a552c8c140064cee859bb1b2185cd92a [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:21:30.779417 9087 raft_consensus.cc:504] T 00000000000000000000000000000000 P a552c8c140064cee859bb1b2185cd92a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a552c8c140064cee859bb1b2185cd92a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34489 } }
I20260929 03:21:30.779505 9087 leader_election.cc:302] T 00000000000000000000000000000000 P a552c8c140064cee859bb1b2185cd92a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a552c8c140064cee859bb1b2185cd92a; no voters:
I20260929 03:21:30.779628 9087 leader_election.cc:288] T 00000000000000000000000000000000 P a552c8c140064cee859bb1b2185cd92a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:21:30.779698 9090 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a552c8c140064cee859bb1b2185cd92a [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:21:30.779857 9090 raft_consensus.cc:686] T 00000000000000000000000000000000 P a552c8c140064cee859bb1b2185cd92a [term 1 LEADER]: Becoming Leader. State: Replica: a552c8c140064cee859bb1b2185cd92a, State: Running, Role: LEADER
I20260929 03:21:30.779899 9087 sys_catalog.cc:564] T 00000000000000000000000000000000 P a552c8c140064cee859bb1b2185cd92a [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:21:30.779944 9090 consensus_queue.cc:237] T 00000000000000000000000000000000 P a552c8c140064cee859bb1b2185cd92a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a552c8c140064cee859bb1b2185cd92a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34489 } }
I20260929 03:21:30.780347 9092 sys_catalog.cc:455] T 00000000000000000000000000000000 P a552c8c140064cee859bb1b2185cd92a [sys.catalog]: SysCatalogTable state changed. Reason: New leader a552c8c140064cee859bb1b2185cd92a. Latest consensus state: current_term: 1 leader_uuid: "a552c8c140064cee859bb1b2185cd92a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a552c8c140064cee859bb1b2185cd92a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34489 } } }
I20260929 03:21:30.780437 9092 sys_catalog.cc:458] T 00000000000000000000000000000000 P a552c8c140064cee859bb1b2185cd92a [sys.catalog]: This master's current role is: LEADER
I20260929 03:21:30.780472 9091 sys_catalog.cc:455] T 00000000000000000000000000000000 P a552c8c140064cee859bb1b2185cd92a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a552c8c140064cee859bb1b2185cd92a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a552c8c140064cee859bb1b2185cd92a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34489 } } }
I20260929 03:21:30.780541 9091 sys_catalog.cc:458] T 00000000000000000000000000000000 P a552c8c140064cee859bb1b2185cd92a [sys.catalog]: This master's current role is: LEADER
I20260929 03:21:30.780689 9097 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:21:30.781243 9097 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:21:30.781592 21193 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuoaHbX/build/release/bin/kudu as pid 9019
I20260929 03:21:30.781713 21193 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790651852476208-21193-0/minicluster-data/master-0/wal/instance
I20260929 03:21:30.781910 21193 external_mini_cluster.cc:1428] Setting key 56acf045ac5bf30efa4d8ba02dfed123
I20260929 03:21:30.783473 9097 catalog_manager.cc:1347] Generated new cluster ID: 9bd96c849d3b495fb8688300eaa73ccb
I20260929 03:21:30.783529 9097 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:21:30.788393 9109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:30.782612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58031 (local address 127.0.0.1:34489)
0929 03:21:30.782807 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0929 03:21:30.782817 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 03:21:30.783065 (+ 248us) server_negotiation.cc:407] Connection header received
0929 03:21:30.783195 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:21:30.783202 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:21:30.783258 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:21:30.783384 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:21:30.783716 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:30.785086 (+ 1370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:30.786155 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:30.786742 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:30.786827 (+ 85us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:21:30.787696 (+ 869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:21:30.787700 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:21:30.787700 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:21:30.787905 (+ 205us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:21:30.787943 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:21:30.787958 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:21:30.788045 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0929 03:21:30.788164 (+ 119us) server_negotiation.cc:299] Negotiation successful
0929 03:21:30.788192 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":75,"threads_started":1}
I20260929 03:21:30.789346 21193 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 03:21:30.796103 9109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:30.790767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48789 (local address 127.0.0.1:34489)
0929 03:21:30.790798 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0929 03:21:30.790800 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:21:30.792815 (+ 2015us) server_negotiation.cc:407] Connection header received
0929 03:21:30.792871 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:21:30.792872 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:21:30.792881 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:21:30.792936 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:21:30.793058 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:30.793778 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:30.794806 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:30.795107 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:30.795168 (+ 61us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:21:30.795853 (+ 685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:21:30.795854 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:21:30.795854 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:21:30.795859 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:21:30.795872 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:21:30.795876 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:21:30.795934 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0929 03:21:30.796013 (+ 79us) server_negotiation.cc:299] Negotiation successful
0929 03:21:30.796024 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260929 03:21:30.797377 9097 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:21:30.798036 9097 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:21:30.800210 9097 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a552c8c140064cee859bb1b2185cd92a: Generated new TSK 0
I20260929 03:21:30.800379 9097 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 03:21:30.805351 21193 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuoaHbX/build/release/bin/kudu with pid 9019
W20260929 03:21:30.809422 9004 connection.cc:535] client connection to 127.0.0.1:34489 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34489): Connection reset by peer (error 104)
2026-09-29T03:21:30Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1308 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790651852476208-21193-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:21:30 dist-test-slave-c9qh krb5kdc[9118](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:21:30 dist-test-slave-c9qh krb5kdc[9118](info): set up 2 sockets
Sep 29 03:21:30 dist-test-slave-c9qh krb5kdc[9118](info): commencing operation
krb5kdc: starting...
W20260929 03:21:31.868616 21193 mini_kdc.cc:121] Time spent starting KDC: real 1.046s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790651852476208-21193-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790651852476208-21193-0/krb5kdc/test-user.keytab.
Sep 29 03:21:31 dist-test-slave-c9qh krb5kdc[9118](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652091, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:21:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:21:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790651852476208-21193-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790651852476208-21193-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 03:21:31.984901 21193 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuoaHbX/build/release/bin/kudu
/tmp/dist-test-taskuoaHbX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790651852476208-21193-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790651852476208-21193-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790651852476208-21193-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790651852476208-21193-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44453
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790651852476208-21193-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44453
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790651852476208-21193-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790651852476208-21193-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790651852476208-21193-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:21:32.067004 9131 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:21:32.067201 9131 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:21:32.067222 9131 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:21:32.068367 9131 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:21:32.068410 9131 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:21:32.068424 9131 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:21:32.068436 9131 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:21:32.068449 9131 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:21:32.070060 9131 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790651852476208-21193-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790651852476208-21193-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44453
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790651852476208-21193-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44453
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790651852476208-21193-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9131
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790651852476208-21193-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:21:32.070312 9131 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:21:32.070554 9131 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:21:32.075445 9139 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:21:32.075460 9137 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:21:32.075493 9136 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:21:32.075827 9131 server_base.cc:1034] running on GCE node
I20260929 03:21:32.076040 9131 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:21:32.076332 9131 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:21:32.077486 9131 hybrid_clock.cc:648] HybridClock initialized: now 1790652092077445 us; error 56 us; skew 500 ppm
Sep 29 03:21:32 dist-test-slave-c9qh krb5kdc[9118](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652092, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:21:32.080025 9131 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 03:21:32.080484 9131 webserver.cc:460] Webserver started at http://127.0.0.1:41335/ using document root <none> and password file <none>
I20260929 03:21:32.080770 9131 fs_manager.cc:359] Metadata directory not provided
I20260929 03:21:32.080816 9131 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:21:32.080960 9131 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:21:32.082465 9131 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790651852476208-21193-0/minicluster-data/master-0/data/instance:
uuid: "76332f95344e44f39c36ec9da22a41a3"
format_stamp: "Formatted at 2026-09-29 03:21:32 on dist-test-slave-c9qh"
server_key: "7cc3a32e482c493b5003f3f833241a74"
server_key_iv: "96459cb60cc2be791d4a499bc24be13d"
server_key_version: "encryptionkey@0"
I20260929 03:21:32.082827 9131 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790651852476208-21193-0/minicluster-data/master-0/wal/instance:
uuid: "76332f95344e44f39c36ec9da22a41a3"
format_stamp: "Formatted at 2026-09-29 03:21:32 on dist-test-slave-c9qh"
server_key: "7cc3a32e482c493b5003f3f833241a74"
server_key_iv: "96459cb60cc2be791d4a499bc24be13d"
server_key_version: "encryptionkey@0"
I20260929 03:21:32.084093 9131 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 03:21:32.084875 9146 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:21:32.085044 9131 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 03:21:32.085134 9131 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790651852476208-21193-0/minicluster-data/master-0/data,/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790651852476208-21193-0/minicluster-data/master-0/wal
uuid: "76332f95344e44f39c36ec9da22a41a3"
format_stamp: "Formatted at 2026-09-29 03:21:32 on dist-test-slave-c9qh"
server_key: "7cc3a32e482c493b5003f3f833241a74"
server_key_iv: "96459cb60cc2be791d4a499bc24be13d"
server_key_version: "encryptionkey@0"
I20260929 03:21:32.085186 9131 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790651852476208-21193-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790651852476208-21193-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790651852476208-21193-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:21:32.088281 9131 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:21:32.090468 9131 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:21:32.090570 9131 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:21:32.094198 9131 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44453
I20260929 03:21:32.094271 9198 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44453 every 8 connection(s)
I20260929 03:21:32.094581 9131 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790651852476208-21193-0/minicluster-data/master-0/data/info.pb
I20260929 03:21:32.095474 9199 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:21:32.097602 9199 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76332f95344e44f39c36ec9da22a41a3: Bootstrap starting.
I20260929 03:21:32.098150 9199 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 76332f95344e44f39c36ec9da22a41a3: Neither blocks nor log segments found. Creating new log.
I20260929 03:21:32.098438 9199 log.cc:824] T 00000000000000000000000000000000 P 76332f95344e44f39c36ec9da22a41a3: Log is configured to *not* fsync() on all Append() calls
I20260929 03:21:32.099141 9199 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76332f95344e44f39c36ec9da22a41a3: No bootstrap required, opened a new log
I20260929 03:21:32.099202 21193 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuoaHbX/build/release/bin/kudu as pid 9131
I20260929 03:21:32.099305 21193 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790651852476208-21193-0/minicluster-data/master-0/wal/instance
I20260929 03:21:32.099500 21193 external_mini_cluster.cc:1428] Setting key 56e98904620663117a29d9d2190e305e
I20260929 03:21:32.101094 9199 raft_consensus.cc:348] T 00000000000000000000000000000000 P 76332f95344e44f39c36ec9da22a41a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76332f95344e44f39c36ec9da22a41a3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44453 } }
I20260929 03:21:32.101251 9199 raft_consensus.cc:374] T 00000000000000000000000000000000 P 76332f95344e44f39c36ec9da22a41a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:21:32.101297 9199 raft_consensus.cc:729] T 00000000000000000000000000000000 P 76332f95344e44f39c36ec9da22a41a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76332f95344e44f39c36ec9da22a41a3, State: Initialized, Role: FOLLOWER
I20260929 03:21:32.101449 9199 consensus_queue.cc:260] T 00000000000000000000000000000000 P 76332f95344e44f39c36ec9da22a41a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76332f95344e44f39c36ec9da22a41a3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44453 } }
I20260929 03:21:32.101541 9199 raft_consensus.cc:388] T 00000000000000000000000000000000 P 76332f95344e44f39c36ec9da22a41a3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:21:32.101585 9199 raft_consensus.cc:482] T 00000000000000000000000000000000 P 76332f95344e44f39c36ec9da22a41a3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:21:32.101635 9199 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 76332f95344e44f39c36ec9da22a41a3 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:21:32.102224 9203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:32.099992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37929 (local address 127.0.0.1:44453)
0929 03:21:32.100167 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0929 03:21:32.100175 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 03:21:32.100213 (+ 38us) server_negotiation.cc:407] Connection header received
0929 03:21:32.101077 (+ 864us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:21:32.101083 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:21:32.101109 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:21:32.101242 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:21:32.101873 (+ 631us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:21:32.101876 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:21:32.101876 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:21:32.101879 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:21:32.101907 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:21:32.101916 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:21:32.101969 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0929 03:21:32.102059 (+ 90us) server_negotiation.cc:299] Negotiation successful
0929 03:21:32.102089 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":72,"threads_started":1}
I20260929 03:21:32.102818 9199 raft_consensus.cc:504] T 00000000000000000000000000000000 P 76332f95344e44f39c36ec9da22a41a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76332f95344e44f39c36ec9da22a41a3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44453 } }
I20260929 03:21:32.102942 9199 leader_election.cc:302] T 00000000000000000000000000000000 P 76332f95344e44f39c36ec9da22a41a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 76332f95344e44f39c36ec9da22a41a3; no voters:
I20260929 03:21:32.103094 9199 leader_election.cc:288] T 00000000000000000000000000000000 P 76332f95344e44f39c36ec9da22a41a3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:21:32.103138 21193 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 03:21:32.103120 9204 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 76332f95344e44f39c36ec9da22a41a3 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:21:32.103267 9204 raft_consensus.cc:686] T 00000000000000000000000000000000 P 76332f95344e44f39c36ec9da22a41a3 [term 1 LEADER]: Becoming Leader. State: Replica: 76332f95344e44f39c36ec9da22a41a3, State: Running, Role: LEADER
I20260929 03:21:32.103340 9199 sys_catalog.cc:564] T 00000000000000000000000000000000 P 76332f95344e44f39c36ec9da22a41a3 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:21:32.103343 9204 consensus_queue.cc:237] T 00000000000000000000000000000000 P 76332f95344e44f39c36ec9da22a41a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76332f95344e44f39c36ec9da22a41a3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44453 } }
I20260929 03:21:32.103657 9204 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76332f95344e44f39c36ec9da22a41a3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 76332f95344e44f39c36ec9da22a41a3. Latest consensus state: current_term: 1 leader_uuid: "76332f95344e44f39c36ec9da22a41a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76332f95344e44f39c36ec9da22a41a3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44453 } } }
I20260929 03:21:32.103742 9204 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76332f95344e44f39c36ec9da22a41a3 [sys.catalog]: This master's current role is: LEADER
I20260929 03:21:32.103696 9206 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76332f95344e44f39c36ec9da22a41a3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "76332f95344e44f39c36ec9da22a41a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76332f95344e44f39c36ec9da22a41a3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44453 } } }
I20260929 03:21:32.103919 9206 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76332f95344e44f39c36ec9da22a41a3 [sys.catalog]: This master's current role is: LEADER
I20260929 03:21:32.103955 9210 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:21:32.104445 9210 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:21:32.105952 9203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:32.104807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58295 (local address 127.0.0.1:44453)
0929 03:21:32.104839 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0929 03:21:32.104841 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:21:32.104979 (+ 138us) server_negotiation.cc:407] Connection header received
0929 03:21:32.105066 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:21:32.105067 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:21:32.105073 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:21:32.105144 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:21:32.105728 (+ 584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:21:32.105729 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:21:32.105730 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:21:32.105734 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:21:32.105748 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:21:32.105753 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:21:32.105816 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0929 03:21:32.105883 (+ 67us) server_negotiation.cc:299] Negotiation successful
0929 03:21:32.105897 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260929 03:21:32.106526 9210 catalog_manager.cc:1347] Generated new cluster ID: 316cd04b1beb472887baa3ced284a260
I20260929 03:21:32.106576 9210 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:21:32.131207 9210 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:21:32.131403 9210 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:21:32.134621 9210 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 76332f95344e44f39c36ec9da22a41a3: Generated new TSK 0
I20260929 03:21:32.134805 9210 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 03:21:32.139055 21193 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuoaHbX/build/release/bin/kudu with pid 9131
2026-09-29T03:21:32Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1333 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790651852476208-21193-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:21:32 dist-test-slave-c9qh krb5kdc[9230](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:21:32 dist-test-slave-c9qh krb5kdc[9230](info): set up 2 sockets
Sep 29 03:21:32 dist-test-slave-c9qh krb5kdc[9230](info): commencing operation
krb5kdc: starting...
W20260929 03:21:33.164876 21193 mini_kdc.cc:121] Time spent starting KDC: real 1.008s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790651852476208-21193-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790651852476208-21193-0/krb5kdc/test-user.keytab.
Sep 29 03:21:33 dist-test-slave-c9qh krb5kdc[9230](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652093, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:21:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:21:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790651852476208-21193-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790651852476208-21193-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 03:21:33.279654 21193 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuoaHbX/build/release/bin/kudu
/tmp/dist-test-taskuoaHbX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790651852476208-21193-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790651852476208-21193-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790651852476208-21193-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790651852476208-21193-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42521
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790651852476208-21193-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42521
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790651852476208-21193-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790651852476208-21193-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790651852476208-21193-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:21:33.359882 9243 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:21:33.360131 9243 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:21:33.360167 9243 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:21:33.361341 9243 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:21:33.361413 9243 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:21:33.361436 9243 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:21:33.361457 9243 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:21:33.361486 9243 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:21:33.363013 9243 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790651852476208-21193-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790651852476208-21193-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42521
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790651852476208-21193-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42521
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790651852476208-21193-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9243
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790651852476208-21193-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:21:33.363306 9243 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:21:33.363550 9243 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:21:33.368446 9251 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:21:33.368417 9249 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:21:33.368456 9248 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:21:33.368692 9243 server_base.cc:1034] running on GCE node
I20260929 03:21:33.368952 9243 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:21:33.369240 9243 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:21:33.370411 9243 hybrid_clock.cc:648] HybridClock initialized: now 1790652093370403 us; error 36 us; skew 500 ppm
Sep 29 03:21:33 dist-test-slave-c9qh krb5kdc[9230](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652093, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:21:33.378286 9243 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 03:21:33.378679 9243 webserver.cc:460] Webserver started at http://127.0.0.1:40709/ using document root <none> and password file <none>
I20260929 03:21:33.378870 9243 fs_manager.cc:359] Metadata directory not provided
I20260929 03:21:33.378911 9243 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:21:33.378997 9243 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:21:33.380036 9243 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790651852476208-21193-0/minicluster-data/master-0/data/instance:
uuid: "b9cab614e1c243bba271ae410372cad2"
format_stamp: "Formatted at 2026-09-29 03:21:33 on dist-test-slave-c9qh"
server_key: "359ac57833e90a8aa9c837493cdf0c7b"
server_key_iv: "24bd18e471e22f1c42a9bf3b6b054db9"
server_key_version: "encryptionkey@0"
I20260929 03:21:33.380334 9243 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790651852476208-21193-0/minicluster-data/master-0/wal/instance:
uuid: "b9cab614e1c243bba271ae410372cad2"
format_stamp: "Formatted at 2026-09-29 03:21:33 on dist-test-slave-c9qh"
server_key: "359ac57833e90a8aa9c837493cdf0c7b"
server_key_iv: "24bd18e471e22f1c42a9bf3b6b054db9"
server_key_version: "encryptionkey@0"
I20260929 03:21:33.381705 9243 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 03:21:33.382434 9258 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:21:33.382622 9243 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260929 03:21:33.382707 9243 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790651852476208-21193-0/minicluster-data/master-0/data,/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790651852476208-21193-0/minicluster-data/master-0/wal
uuid: "b9cab614e1c243bba271ae410372cad2"
format_stamp: "Formatted at 2026-09-29 03:21:33 on dist-test-slave-c9qh"
server_key: "359ac57833e90a8aa9c837493cdf0c7b"
server_key_iv: "24bd18e471e22f1c42a9bf3b6b054db9"
server_key_version: "encryptionkey@0"
I20260929 03:21:33.382762 9243 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790651852476208-21193-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790651852476208-21193-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790651852476208-21193-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:21:33.396430 9243 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:21:33.398613 9243 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:21:33.398718 9243 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:21:33.402498 9243 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42521
I20260929 03:21:33.402562 9310 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42521 every 8 connection(s)
I20260929 03:21:33.402880 9243 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790651852476208-21193-0/minicluster-data/master-0/data/info.pb
I20260929 03:21:33.403731 9311 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:21:33.403995 21193 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuoaHbX/build/release/bin/kudu as pid 9243
I20260929 03:21:33.404101 21193 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790651852476208-21193-0/minicluster-data/master-0/wal/instance
I20260929 03:21:33.404299 21193 external_mini_cluster.cc:1428] Setting key 1fb0ef5219c320a083e21d6316f52651
I20260929 03:21:33.407218 9311 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9cab614e1c243bba271ae410372cad2: Bootstrap starting.
I20260929 03:21:33.407876 9311 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b9cab614e1c243bba271ae410372cad2: Neither blocks nor log segments found. Creating new log.
I20260929 03:21:33.408170 9311 log.cc:824] T 00000000000000000000000000000000 P b9cab614e1c243bba271ae410372cad2: Log is configured to *not* fsync() on all Append() calls
I20260929 03:21:33.408890 9311 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9cab614e1c243bba271ae410372cad2: No bootstrap required, opened a new log
I20260929 03:21:33.410784 9311 raft_consensus.cc:348] T 00000000000000000000000000000000 P b9cab614e1c243bba271ae410372cad2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9cab614e1c243bba271ae410372cad2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42521 } }
I20260929 03:21:33.410935 9311 raft_consensus.cc:374] T 00000000000000000000000000000000 P b9cab614e1c243bba271ae410372cad2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:21:33.410980 9311 raft_consensus.cc:729] T 00000000000000000000000000000000 P b9cab614e1c243bba271ae410372cad2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9cab614e1c243bba271ae410372cad2, State: Initialized, Role: FOLLOWER
Sep 29 03:21:33 dist-test-slave-c9qh krb5kdc[9230](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652093, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 03:21:33.411108 9311 consensus_queue.cc:260] T 00000000000000000000000000000000 P b9cab614e1c243bba271ae410372cad2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9cab614e1c243bba271ae410372cad2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42521 } }
I20260929 03:21:33.411201 9311 raft_consensus.cc:388] T 00000000000000000000000000000000 P b9cab614e1c243bba271ae410372cad2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:21:33.411255 9311 raft_consensus.cc:482] T 00000000000000000000000000000000 P b9cab614e1c243bba271ae410372cad2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:21:33.411314 9311 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b9cab614e1c243bba271ae410372cad2 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:21:33.411958 9311 raft_consensus.cc:504] T 00000000000000000000000000000000 P b9cab614e1c243bba271ae410372cad2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9cab614e1c243bba271ae410372cad2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42521 } }
I20260929 03:21:33.412082 9311 leader_election.cc:302] T 00000000000000000000000000000000 P b9cab614e1c243bba271ae410372cad2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b9cab614e1c243bba271ae410372cad2; no voters:
I20260929 03:21:33.412230 9311 leader_election.cc:288] T 00000000000000000000000000000000 P b9cab614e1c243bba271ae410372cad2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:21:33.412261 9316 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b9cab614e1c243bba271ae410372cad2 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:21:33.412374 9316 raft_consensus.cc:686] T 00000000000000000000000000000000 P b9cab614e1c243bba271ae410372cad2 [term 1 LEADER]: Becoming Leader. State: Replica: b9cab614e1c243bba271ae410372cad2, State: Running, Role: LEADER
I20260929 03:21:33.412493 9311 sys_catalog.cc:564] T 00000000000000000000000000000000 P b9cab614e1c243bba271ae410372cad2 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:21:33.412503 9316 consensus_queue.cc:237] T 00000000000000000000000000000000 P b9cab614e1c243bba271ae410372cad2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9cab614e1c243bba271ae410372cad2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42521 } }
I20260929 03:21:33.412847 9316 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9cab614e1c243bba271ae410372cad2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b9cab614e1c243bba271ae410372cad2. Latest consensus state: current_term: 1 leader_uuid: "b9cab614e1c243bba271ae410372cad2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9cab614e1c243bba271ae410372cad2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42521 } } }
I20260929 03:21:33.412961 9316 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9cab614e1c243bba271ae410372cad2 [sys.catalog]: This master's current role is: LEADER
I20260929 03:21:33.412981 9317 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9cab614e1c243bba271ae410372cad2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b9cab614e1c243bba271ae410372cad2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9cab614e1c243bba271ae410372cad2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42521 } } }
I20260929 03:21:33.413142 9317 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9cab614e1c243bba271ae410372cad2 [sys.catalog]: This master's current role is: LEADER
I20260929 03:21:33.413223 9323 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:21:33.413715 9323 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:21:33.415539 9314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:33.404855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41287 (local address 127.0.0.1:42521)
0929 03:21:33.405049 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0929 03:21:33.405058 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 03:21:33.405090 (+ 32us) server_negotiation.cc:407] Connection header received
0929 03:21:33.405551 (+ 461us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:21:33.405560 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:21:33.405640 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:21:33.405796 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:21:33.406469 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:33.407833 (+ 1364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:33.408656 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:33.409242 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:33.409362 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:21:33.411639 (+ 2277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:21:33.411643 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:21:33.411644 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:21:33.411655 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:21:33.413806 (+ 2151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:21:33.414291 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:21:33.414293 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:21:33.414294 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:21:33.414408 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:21:33.414637 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:21:33.414638 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:21:33.414639 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:21:33.414945 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:21:33.415075 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0929 03:21:33.415222 (+ 147us) server_negotiation.cc:299] Negotiation successful
0929 03:21:33.415354 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":73,"threads_started":1}
I20260929 03:21:33.415844 9323 catalog_manager.cc:1347] Generated new cluster ID: e7b45a2a071a462d8a471ce9253e53b4
I20260929 03:21:33.415923 9323 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:21:33.416355 21193 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 03:21:33.423604 9314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:33.417791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53247 (local address 127.0.0.1:42521)
0929 03:21:33.417825 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0929 03:21:33.417827 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:21:33.417935 (+ 108us) server_negotiation.cc:407] Connection header received
0929 03:21:33.418007 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:21:33.418008 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:21:33.418017 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:21:33.418073 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:21:33.418570 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:33.419008 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:33.419677 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:33.419886 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:33.419967 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:21:33.421168 (+ 1201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:21:33.421169 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:21:33.421169 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:21:33.421176 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:21:33.422490 (+ 1314us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:21:33.422863 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:21:33.422864 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:21:33.422865 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:21:33.422917 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:21:33.423119 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:21:33.423120 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:21:33.423120 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:21:33.423227 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:21:33.423325 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0929 03:21:33.423426 (+ 101us) server_negotiation.cc:299] Negotiation successful
0929 03:21:33.423499 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260929 03:21:33.429232 9323 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:21:33.429699 9323 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:21:33.438038 9323 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b9cab614e1c243bba271ae410372cad2: Generated new TSK 0
I20260929 03:21:33.438246 9323 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 03:21:33.442251 21193 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuoaHbX/build/release/bin/kudu with pid 9243
2026-09-29T03:21:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1303 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790651852476208-21193-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:21:33 dist-test-slave-c9qh krb5kdc[9342](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:21:33 dist-test-slave-c9qh krb5kdc[9342](info): set up 2 sockets
Sep 29 03:21:33 dist-test-slave-c9qh krb5kdc[9342](info): commencing operation
krb5kdc: starting...
W20260929 03:21:34.464793 21193 mini_kdc.cc:121] Time spent starting KDC: real 1.004s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790651852476208-21193-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790651852476208-21193-0/krb5kdc/test-user.keytab.
Sep 29 03:21:34 dist-test-slave-c9qh krb5kdc[9342](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652094, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:21:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:21:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790651852476208-21193-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790651852476208-21193-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 03:21:34.581386 21193 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuoaHbX/build/release/bin/kudu
/tmp/dist-test-taskuoaHbX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790651852476208-21193-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790651852476208-21193-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790651852476208-21193-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790651852476208-21193-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35981
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790651852476208-21193-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35981
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790651852476208-21193-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790651852476208-21193-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790651852476208-21193-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:21:34.660545 9355 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:21:34.660872 9355 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:21:34.660913 9355 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:21:34.662794 9355 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:21:34.662846 9355 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:21:34.662876 9355 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:21:34.662901 9355 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:21:34.662925 9355 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:21:34.664743 9355 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790651852476208-21193-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790651852476208-21193-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35981
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790651852476208-21193-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35981
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790651852476208-21193-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9355
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790651852476208-21193-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:21:34.664949 9355 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:21:34.665110 9355 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:21:34.669894 9363 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:21:34.669884 9361 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:21:34.669972 9355 server_base.cc:1034] running on GCE node
W20260929 03:21:34.669884 9360 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:21:34.670305 9355 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:21:34.670624 9355 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:21:34.671783 9355 hybrid_clock.cc:648] HybridClock initialized: now 1790652094671761 us; error 37 us; skew 500 ppm
Sep 29 03:21:34 dist-test-slave-c9qh krb5kdc[9342](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652094, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:21:34.674211 9355 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 03:21:34.674672 9355 webserver.cc:460] Webserver started at http://127.0.0.1:44701/ using document root <none> and password file <none>
I20260929 03:21:34.674892 9355 fs_manager.cc:359] Metadata directory not provided
I20260929 03:21:34.674960 9355 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:21:34.675107 9355 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:21:34.676167 9355 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790651852476208-21193-0/minicluster-data/master-0/data/instance:
uuid: "52d55b8f772e4c38adfbb13a4461736e"
format_stamp: "Formatted at 2026-09-29 03:21:34 on dist-test-slave-c9qh"
server_key: "11d6818aa83ad16da94163f4f2492990"
server_key_iv: "74ae216449909777fcced1dae9d2af92"
server_key_version: "encryptionkey@0"
I20260929 03:21:34.676535 9355 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790651852476208-21193-0/minicluster-data/master-0/wal/instance:
uuid: "52d55b8f772e4c38adfbb13a4461736e"
format_stamp: "Formatted at 2026-09-29 03:21:34 on dist-test-slave-c9qh"
server_key: "11d6818aa83ad16da94163f4f2492990"
server_key_iv: "74ae216449909777fcced1dae9d2af92"
server_key_version: "encryptionkey@0"
I20260929 03:21:34.677846 9355 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260929 03:21:34.678519 9370 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:21:34.678699 9355 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 03:21:34.678792 9355 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790651852476208-21193-0/minicluster-data/master-0/data,/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790651852476208-21193-0/minicluster-data/master-0/wal
uuid: "52d55b8f772e4c38adfbb13a4461736e"
format_stamp: "Formatted at 2026-09-29 03:21:34 on dist-test-slave-c9qh"
server_key: "11d6818aa83ad16da94163f4f2492990"
server_key_iv: "74ae216449909777fcced1dae9d2af92"
server_key_version: "encryptionkey@0"
I20260929 03:21:34.678893 9355 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790651852476208-21193-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790651852476208-21193-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790651852476208-21193-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:21:34.687387 9355 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:21:34.689266 9355 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:21:34.689428 9355 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:21:34.693320 9355 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35981
I20260929 03:21:34.693377 9422 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35981 every 8 connection(s)
I20260929 03:21:34.693725 9355 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790651852476208-21193-0/minicluster-data/master-0/data/info.pb
I20260929 03:21:34.694525 9423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:21:34.695688 21193 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuoaHbX/build/release/bin/kudu as pid 9355
I20260929 03:21:34.695822 21193 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790651852476208-21193-0/minicluster-data/master-0/wal/instance
I20260929 03:21:34.696014 21193 external_mini_cluster.cc:1428] Setting key 3bfcaba08210fb47836b49ded86303ba
I20260929 03:21:34.697240 9423 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52d55b8f772e4c38adfbb13a4461736e: Bootstrap starting.
I20260929 03:21:34.697934 9423 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 52d55b8f772e4c38adfbb13a4461736e: Neither blocks nor log segments found. Creating new log.
I20260929 03:21:34.698236 9423 log.cc:824] T 00000000000000000000000000000000 P 52d55b8f772e4c38adfbb13a4461736e: Log is configured to *not* fsync() on all Append() calls
I20260929 03:21:34.698978 9423 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52d55b8f772e4c38adfbb13a4461736e: No bootstrap required, opened a new log
I20260929 03:21:34.701102 9423 raft_consensus.cc:348] T 00000000000000000000000000000000 P 52d55b8f772e4c38adfbb13a4461736e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52d55b8f772e4c38adfbb13a4461736e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35981 } }
I20260929 03:21:34.701225 9423 raft_consensus.cc:374] T 00000000000000000000000000000000 P 52d55b8f772e4c38adfbb13a4461736e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:21:34.701272 9423 raft_consensus.cc:729] T 00000000000000000000000000000000 P 52d55b8f772e4c38adfbb13a4461736e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52d55b8f772e4c38adfbb13a4461736e, State: Initialized, Role: FOLLOWER
I20260929 03:21:34.701414 9423 consensus_queue.cc:260] T 00000000000000000000000000000000 P 52d55b8f772e4c38adfbb13a4461736e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52d55b8f772e4c38adfbb13a4461736e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35981 } }
I20260929 03:21:34.701504 9423 raft_consensus.cc:388] T 00000000000000000000000000000000 P 52d55b8f772e4c38adfbb13a4461736e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:21:34.701536 9423 raft_consensus.cc:482] T 00000000000000000000000000000000 P 52d55b8f772e4c38adfbb13a4461736e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:21:34.701582 9423 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 52d55b8f772e4c38adfbb13a4461736e [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:21:34.701995 9426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:34.696532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57803 (local address 127.0.0.1:35981)
0929 03:21:34.696744 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0929 03:21:34.696754 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 03:21:34.696788 (+ 34us) server_negotiation.cc:407] Connection header received
0929 03:21:34.697335 (+ 547us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:21:34.697344 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:21:34.697405 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:21:34.697558 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:21:34.697723 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:34.699107 (+ 1384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:34.699868 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:34.700483 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:34.700616 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:21:34.701327 (+ 711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:21:34.701331 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:21:34.701331 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:21:34.701531 (+ 200us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:21:34.701571 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:21:34.701586 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:21:34.701653 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0929 03:21:34.701720 (+ 67us) server_negotiation.cc:299] Negotiation successful
0929 03:21:34.701781 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":76,"threads_started":1}
I20260929 03:21:34.702231 9423 raft_consensus.cc:504] T 00000000000000000000000000000000 P 52d55b8f772e4c38adfbb13a4461736e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52d55b8f772e4c38adfbb13a4461736e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35981 } }
I20260929 03:21:34.702356 9423 leader_election.cc:302] T 00000000000000000000000000000000 P 52d55b8f772e4c38adfbb13a4461736e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 52d55b8f772e4c38adfbb13a4461736e; no voters:
I20260929 03:21:34.702525 9423 leader_election.cc:288] T 00000000000000000000000000000000 P 52d55b8f772e4c38adfbb13a4461736e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:21:34.702548 9428 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 52d55b8f772e4c38adfbb13a4461736e [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:21:34.702649 9428 raft_consensus.cc:686] T 00000000000000000000000000000000 P 52d55b8f772e4c38adfbb13a4461736e [term 1 LEADER]: Becoming Leader. State: Replica: 52d55b8f772e4c38adfbb13a4461736e, State: Running, Role: LEADER
I20260929 03:21:34.702721 9428 consensus_queue.cc:237] T 00000000000000000000000000000000 P 52d55b8f772e4c38adfbb13a4461736e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52d55b8f772e4c38adfbb13a4461736e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35981 } }
I20260929 03:21:34.702759 9423 sys_catalog.cc:564] T 00000000000000000000000000000000 P 52d55b8f772e4c38adfbb13a4461736e [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:21:34.702836 21193 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 03:21:34.703034 9428 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52d55b8f772e4c38adfbb13a4461736e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 52d55b8f772e4c38adfbb13a4461736e. Latest consensus state: current_term: 1 leader_uuid: "52d55b8f772e4c38adfbb13a4461736e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52d55b8f772e4c38adfbb13a4461736e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35981 } } }
I20260929 03:21:34.703109 9428 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52d55b8f772e4c38adfbb13a4461736e [sys.catalog]: This master's current role is: LEADER
I20260929 03:21:34.703316 9434 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:21:34.704003 9434 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:21:34.704412 9429 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52d55b8f772e4c38adfbb13a4461736e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "52d55b8f772e4c38adfbb13a4461736e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52d55b8f772e4c38adfbb13a4461736e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35981 } } }
I20260929 03:21:34.705564 9429 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52d55b8f772e4c38adfbb13a4461736e [sys.catalog]: This master's current role is: LEADER
I20260929 03:21:34.705639 9434 catalog_manager.cc:1347] Generated new cluster ID: d4659437a82f40e785ff7a30e905317a
I20260929 03:21:34.705681 9434 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:21:34.707360 9426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:34.704493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52425 (local address 127.0.0.1:35981)
0929 03:21:34.704531 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0929 03:21:34.704532 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 03:21:34.704540 (+ 8us) server_negotiation.cc:407] Connection header received
0929 03:21:34.704560 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:21:34.704561 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:21:34.704570 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:21:34.704648 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:21:34.704820 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:34.705453 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:34.706171 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:34.706433 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:34.706511 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:21:34.707148 (+ 637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:21:34.707149 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:21:34.707149 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:21:34.707154 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:21:34.707165 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:21:34.707169 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:21:34.707211 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0929 03:21:34.707264 (+ 53us) server_negotiation.cc:299] Negotiation successful
0929 03:21:34.707285 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260929 03:21:34.727224 9434 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:21:34.727766 9434 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:21:34.731966 9434 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 52d55b8f772e4c38adfbb13a4461736e: Generated new TSK 0
I20260929 03:21:34.732143 9434 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 03:21:34.739285 21193 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuoaHbX/build/release/bin/kudu with pid 9355
2026-09-29T03:21:34Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1296 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790651852476208-21193-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:21:34 dist-test-slave-c9qh krb5kdc[9454](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:21:34 dist-test-slave-c9qh krb5kdc[9454](info): set up 2 sockets
Sep 29 03:21:34 dist-test-slave-c9qh krb5kdc[9454](info): commencing operation
krb5kdc: starting...
W20260929 03:21:35.769238 21193 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790651852476208-21193-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790651852476208-21193-0/krb5kdc/test-user.keytab.
Sep 29 03:21:35 dist-test-slave-c9qh krb5kdc[9454](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652095, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:21:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:21:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790651852476208-21193-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790651852476208-21193-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 03:21:35.883787 21193 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuoaHbX/build/release/bin/kudu
/tmp/dist-test-taskuoaHbX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790651852476208-21193-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790651852476208-21193-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790651852476208-21193-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790651852476208-21193-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32875
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790651852476208-21193-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:32875
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790651852476208-21193-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790651852476208-21193-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790651852476208-21193-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:21:35.971454 9467 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:21:35.971668 9467 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:21:35.971690 9467 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:21:35.972888 9467 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:21:35.972929 9467 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:21:35.972944 9467 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:21:35.972956 9467 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:21:35.973001 9467 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:21:35.974484 9467 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790651852476208-21193-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790651852476208-21193-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:32875
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790651852476208-21193-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32875
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790651852476208-21193-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9467
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790651852476208-21193-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:21:35.974717 9467 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:21:35.974946 9467 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:21:35.979912 9475 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:21:35.980038 9467 server_base.cc:1034] running on GCE node
W20260929 03:21:35.979925 9472 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:21:35.979959 9473 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:21:35.980521 9467 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:21:35.980918 9467 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:21:35.982067 9467 hybrid_clock.cc:648] HybridClock initialized: now 1790652095982045 us; error 34 us; skew 500 ppm
Sep 29 03:21:35 dist-test-slave-c9qh krb5kdc[9454](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652095, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:21:35.984586 9467 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 03:21:35.985112 9467 webserver.cc:460] Webserver started at http://127.0.0.1:35613/ using document root <none> and password file <none>
I20260929 03:21:35.985370 9467 fs_manager.cc:359] Metadata directory not provided
I20260929 03:21:35.985421 9467 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:21:35.985599 9467 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:21:35.986814 9467 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790651852476208-21193-0/minicluster-data/master-0/data/instance:
uuid: "5fdc47db4a9d43069c361d9d3cea1c55"
format_stamp: "Formatted at 2026-09-29 03:21:35 on dist-test-slave-c9qh"
server_key: "1f919427e4e3ef7e500c0417684b9325"
server_key_iv: "356dbafd70152cb68c616d2590ca191d"
server_key_version: "encryptionkey@0"
I20260929 03:21:35.987293 9467 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790651852476208-21193-0/minicluster-data/master-0/wal/instance:
uuid: "5fdc47db4a9d43069c361d9d3cea1c55"
format_stamp: "Formatted at 2026-09-29 03:21:35 on dist-test-slave-c9qh"
server_key: "1f919427e4e3ef7e500c0417684b9325"
server_key_iv: "356dbafd70152cb68c616d2590ca191d"
server_key_version: "encryptionkey@0"
I20260929 03:21:35.988826 9467 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.002s
I20260929 03:21:35.989562 9482 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:21:35.989739 9467 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260929 03:21:35.989845 9467 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790651852476208-21193-0/minicluster-data/master-0/data,/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790651852476208-21193-0/minicluster-data/master-0/wal
uuid: "5fdc47db4a9d43069c361d9d3cea1c55"
format_stamp: "Formatted at 2026-09-29 03:21:35 on dist-test-slave-c9qh"
server_key: "1f919427e4e3ef7e500c0417684b9325"
server_key_iv: "356dbafd70152cb68c616d2590ca191d"
server_key_version: "encryptionkey@0"
I20260929 03:21:35.989936 9467 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790651852476208-21193-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790651852476208-21193-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790651852476208-21193-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:21:35.993561 9467 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:21:35.995460 9467 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:21:35.995625 9467 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:21:35.999812 9467 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:32875
I20260929 03:21:35.999845 9534 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:32875 every 8 connection(s)
I20260929 03:21:36.000182 9467 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790651852476208-21193-0/minicluster-data/master-0/data/info.pb
I20260929 03:21:36.001076 9535 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:21:36.003209 9535 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5fdc47db4a9d43069c361d9d3cea1c55: Bootstrap starting.
I20260929 03:21:36.003707 9535 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5fdc47db4a9d43069c361d9d3cea1c55: Neither blocks nor log segments found. Creating new log.
I20260929 03:21:36.003942 9535 log.cc:824] T 00000000000000000000000000000000 P 5fdc47db4a9d43069c361d9d3cea1c55: Log is configured to *not* fsync() on all Append() calls
I20260929 03:21:36.004530 9535 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5fdc47db4a9d43069c361d9d3cea1c55: No bootstrap required, opened a new log
I20260929 03:21:36.006090 9535 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5fdc47db4a9d43069c361d9d3cea1c55 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fdc47db4a9d43069c361d9d3cea1c55" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32875 } }
I20260929 03:21:36.006218 9535 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5fdc47db4a9d43069c361d9d3cea1c55 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:21:36.006242 9535 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5fdc47db4a9d43069c361d9d3cea1c55 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fdc47db4a9d43069c361d9d3cea1c55, State: Initialized, Role: FOLLOWER
I20260929 03:21:36.006330 9535 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5fdc47db4a9d43069c361d9d3cea1c55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fdc47db4a9d43069c361d9d3cea1c55" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32875 } }
I20260929 03:21:36.006381 9535 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5fdc47db4a9d43069c361d9d3cea1c55 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:21:36.006402 9535 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5fdc47db4a9d43069c361d9d3cea1c55 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:21:36.006479 9535 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5fdc47db4a9d43069c361d9d3cea1c55 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:21:36.006987 9535 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5fdc47db4a9d43069c361d9d3cea1c55 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fdc47db4a9d43069c361d9d3cea1c55" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32875 } }
I20260929 03:21:36.007107 9535 leader_election.cc:302] T 00000000000000000000000000000000 P 5fdc47db4a9d43069c361d9d3cea1c55 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5fdc47db4a9d43069c361d9d3cea1c55; no voters:
I20260929 03:21:36.007282 9535 leader_election.cc:288] T 00000000000000000000000000000000 P 5fdc47db4a9d43069c361d9d3cea1c55 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:21:36.007326 9538 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5fdc47db4a9d43069c361d9d3cea1c55 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:21:36.007495 9538 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5fdc47db4a9d43069c361d9d3cea1c55 [term 1 LEADER]: Becoming Leader. State: Replica: 5fdc47db4a9d43069c361d9d3cea1c55, State: Running, Role: LEADER
I20260929 03:21:36.007536 9535 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5fdc47db4a9d43069c361d9d3cea1c55 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:21:36.007609 9538 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5fdc47db4a9d43069c361d9d3cea1c55 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fdc47db4a9d43069c361d9d3cea1c55" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32875 } }
I20260929 03:21:36.008037 9539 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5fdc47db4a9d43069c361d9d3cea1c55 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5fdc47db4a9d43069c361d9d3cea1c55" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fdc47db4a9d43069c361d9d3cea1c55" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32875 } } }
I20260929 03:21:36.008059 9540 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5fdc47db4a9d43069c361d9d3cea1c55 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5fdc47db4a9d43069c361d9d3cea1c55. Latest consensus state: current_term: 1 leader_uuid: "5fdc47db4a9d43069c361d9d3cea1c55" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fdc47db4a9d43069c361d9d3cea1c55" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32875 } } }
I20260929 03:21:36.008210 9540 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5fdc47db4a9d43069c361d9d3cea1c55 [sys.catalog]: This master's current role is: LEADER
I20260929 03:21:36.008198 9539 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5fdc47db4a9d43069c361d9d3cea1c55 [sys.catalog]: This master's current role is: LEADER
I20260929 03:21:36.008589 21193 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuoaHbX/build/release/bin/kudu as pid 9467
I20260929 03:21:36.008723 21193 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790651852476208-21193-0/minicluster-data/master-0/wal/instance
I20260929 03:21:36.008867 21193 external_mini_cluster.cc:1428] Setting key 35bbbe0dcec9c5547a262e3d4261b90f
I20260929 03:21:36.009282 9543 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:21:36.009841 9543 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:21:36.011821 9543 catalog_manager.cc:1347] Generated new cluster ID: b78daded96e44aa79a59b66b0835bf61
I20260929 03:21:36.011893 9543 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:21:36.012005 9550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:36.009306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37837 (local address 127.0.0.1:32875)
0929 03:21:36.010196 (+ 890us) server_negotiation.cc:206] Beginning negotiation
0929 03:21:36.010206 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 03:21:36.010239 (+ 33us) server_negotiation.cc:407] Connection header received
0929 03:21:36.010512 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:21:36.010517 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:21:36.010565 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:21:36.010710 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:21:36.011500 (+ 790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:21:36.011502 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:21:36.011503 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:21:36.011506 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:21:36.011538 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:21:36.011549 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:21:36.011627 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0929 03:21:36.011703 (+ 76us) server_negotiation.cc:299] Negotiation successful
0929 03:21:36.011746 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":841,"thread_start_us":73,"threads_started":1}
I20260929 03:21:36.012858 21193 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 03:21:36.021469 9543 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:21:36.021651 9543 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:21:36.021795 9550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:36.016990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39739 (local address 127.0.0.1:32875)
0929 03:21:36.017050 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0929 03:21:36.017052 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:21:36.020743 (+ 3691us) server_negotiation.cc:407] Connection header received
0929 03:21:36.020882 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:21:36.020883 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:21:36.020891 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:21:36.020970 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:21:36.021532 (+ 562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:21:36.021534 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:21:36.021534 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:21:36.021539 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:21:36.021555 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:21:36.021560 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:21:36.021624 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0929 03:21:36.021713 (+ 89us) server_negotiation.cc:299] Negotiation successful
0929 03:21:36.021731 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260929 03:21:36.026716 9543 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5fdc47db4a9d43069c361d9d3cea1c55: Generated new TSK 0
I20260929 03:21:36.026909 9543 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 03:21:36.032234 21193 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuoaHbX/build/release/bin/kudu with pid 9467
2026-09-29T03:21:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1293 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790651852476208-21193-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:21:36 dist-test-slave-c9qh krb5kdc[9566](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:21:36 dist-test-slave-c9qh krb5kdc[9566](info): set up 2 sockets
Sep 29 03:21:36 dist-test-slave-c9qh krb5kdc[9566](info): commencing operation
krb5kdc: starting...
W20260929 03:21:37.084834 21193 mini_kdc.cc:121] Time spent starting KDC: real 1.034s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790651852476208-21193-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790651852476208-21193-0/krb5kdc/test-user.keytab.
Sep 29 03:21:37 dist-test-slave-c9qh krb5kdc[9566](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652097, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:21:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:21:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790651852476208-21193-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790651852476208-21193-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 03:21:37.199499 21193 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuoaHbX/build/release/bin/kudu
/tmp/dist-test-taskuoaHbX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790651852476208-21193-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790651852476208-21193-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790651852476208-21193-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790651852476208-21193-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41915
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790651852476208-21193-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41915
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790651852476208-21193-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790651852476208-21193-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790651852476208-21193-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:21:37.280267 9579 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:21:37.280517 9579 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:21:37.280557 9579 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:21:37.281724 9579 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:21:37.281791 9579 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:21:37.281816 9579 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:21:37.281843 9579 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:21:37.281869 9579 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:21:37.283303 9579 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790651852476208-21193-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790651852476208-21193-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41915
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790651852476208-21193-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41915
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790651852476208-21193-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9579
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790651852476208-21193-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:21:37.283571 9579 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:21:37.283798 9579 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:21:37.288604 9585 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:21:37.288604 9587 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:21:37.288758 9579 server_base.cc:1034] running on GCE node
W20260929 03:21:37.288610 9584 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:21:37.289055 9579 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:21:37.289337 9579 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:21:37.290501 9579 hybrid_clock.cc:648] HybridClock initialized: now 1790652097290497 us; error 37 us; skew 500 ppm
Sep 29 03:21:37 dist-test-slave-c9qh krb5kdc[9566](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652097, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:21:37.292932 9579 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 03:21:37.293385 9579 webserver.cc:460] Webserver started at http://127.0.0.1:40845/ using document root <none> and password file <none>
I20260929 03:21:37.293612 9579 fs_manager.cc:359] Metadata directory not provided
I20260929 03:21:37.293678 9579 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:21:37.293828 9579 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:21:37.294832 9579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790651852476208-21193-0/minicluster-data/master-0/data/instance:
uuid: "f45e3e7960284dd0b2a6e77e6054f6f5"
format_stamp: "Formatted at 2026-09-29 03:21:37 on dist-test-slave-c9qh"
server_key: "804dc7cb1d0af92d21c7706c7f7847da"
server_key_iv: "bd9a4e30442f2b3366e8d14469ddc35c"
server_key_version: "encryptionkey@0"
I20260929 03:21:37.295200 9579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790651852476208-21193-0/minicluster-data/master-0/wal/instance:
uuid: "f45e3e7960284dd0b2a6e77e6054f6f5"
format_stamp: "Formatted at 2026-09-29 03:21:37 on dist-test-slave-c9qh"
server_key: "804dc7cb1d0af92d21c7706c7f7847da"
server_key_iv: "bd9a4e30442f2b3366e8d14469ddc35c"
server_key_version: "encryptionkey@0"
I20260929 03:21:37.296530 9579 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260929 03:21:37.297338 9594 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:21:37.297549 9579 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260929 03:21:37.297662 9579 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790651852476208-21193-0/minicluster-data/master-0/data,/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790651852476208-21193-0/minicluster-data/master-0/wal
uuid: "f45e3e7960284dd0b2a6e77e6054f6f5"
format_stamp: "Formatted at 2026-09-29 03:21:37 on dist-test-slave-c9qh"
server_key: "804dc7cb1d0af92d21c7706c7f7847da"
server_key_iv: "bd9a4e30442f2b3366e8d14469ddc35c"
server_key_version: "encryptionkey@0"
I20260929 03:21:37.297746 9579 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790651852476208-21193-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790651852476208-21193-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790651852476208-21193-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:21:37.312181 9579 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:21:37.314034 9579 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:21:37.314199 9579 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:21:37.318130 9579 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41915
I20260929 03:21:37.318161 9646 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41915 every 8 connection(s)
I20260929 03:21:37.318506 9579 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790651852476208-21193-0/minicluster-data/master-0/data/info.pb
I20260929 03:21:37.319300 9647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:21:37.321624 9647 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f45e3e7960284dd0b2a6e77e6054f6f5: Bootstrap starting.
I20260929 03:21:37.322202 9647 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f45e3e7960284dd0b2a6e77e6054f6f5: Neither blocks nor log segments found. Creating new log.
I20260929 03:21:37.322427 9647 log.cc:824] T 00000000000000000000000000000000 P f45e3e7960284dd0b2a6e77e6054f6f5: Log is configured to *not* fsync() on all Append() calls
I20260929 03:21:37.323139 9647 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f45e3e7960284dd0b2a6e77e6054f6f5: No bootstrap required, opened a new log
I20260929 03:21:37.323884 21193 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuoaHbX/build/release/bin/kudu as pid 9579
I20260929 03:21:37.324014 21193 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790651852476208-21193-0/minicluster-data/master-0/wal/instance
I20260929 03:21:37.324208 21193 external_mini_cluster.cc:1428] Setting key aa67ede13720d3070bed5a4655526df0
I20260929 03:21:37.325243 9647 raft_consensus.cc:348] T 00000000000000000000000000000000 P f45e3e7960284dd0b2a6e77e6054f6f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f45e3e7960284dd0b2a6e77e6054f6f5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41915 } }
I20260929 03:21:37.325414 9647 raft_consensus.cc:374] T 00000000000000000000000000000000 P f45e3e7960284dd0b2a6e77e6054f6f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:21:37.325452 9647 raft_consensus.cc:729] T 00000000000000000000000000000000 P f45e3e7960284dd0b2a6e77e6054f6f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f45e3e7960284dd0b2a6e77e6054f6f5, State: Initialized, Role: FOLLOWER
I20260929 03:21:37.325618 9647 consensus_queue.cc:260] T 00000000000000000000000000000000 P f45e3e7960284dd0b2a6e77e6054f6f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f45e3e7960284dd0b2a6e77e6054f6f5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41915 } }
I20260929 03:21:37.325698 9647 raft_consensus.cc:388] T 00000000000000000000000000000000 P f45e3e7960284dd0b2a6e77e6054f6f5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:21:37.325721 9647 raft_consensus.cc:482] T 00000000000000000000000000000000 P f45e3e7960284dd0b2a6e77e6054f6f5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:21:37.325771 9647 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f45e3e7960284dd0b2a6e77e6054f6f5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:21:37.326768 9647 raft_consensus.cc:504] T 00000000000000000000000000000000 P f45e3e7960284dd0b2a6e77e6054f6f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f45e3e7960284dd0b2a6e77e6054f6f5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41915 } }
I20260929 03:21:37.326881 9647 leader_election.cc:302] T 00000000000000000000000000000000 P f45e3e7960284dd0b2a6e77e6054f6f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f45e3e7960284dd0b2a6e77e6054f6f5; no voters:
I20260929 03:21:37.327024 9647 leader_election.cc:288] T 00000000000000000000000000000000 P f45e3e7960284dd0b2a6e77e6054f6f5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:21:37.327102 9652 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f45e3e7960284dd0b2a6e77e6054f6f5 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:21:37.327308 9652 raft_consensus.cc:686] T 00000000000000000000000000000000 P f45e3e7960284dd0b2a6e77e6054f6f5 [term 1 LEADER]: Becoming Leader. State: Replica: f45e3e7960284dd0b2a6e77e6054f6f5, State: Running, Role: LEADER
I20260929 03:21:37.327328 9647 sys_catalog.cc:564] T 00000000000000000000000000000000 P f45e3e7960284dd0b2a6e77e6054f6f5 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:21:37.328055 9652 consensus_queue.cc:237] T 00000000000000000000000000000000 P f45e3e7960284dd0b2a6e77e6054f6f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f45e3e7960284dd0b2a6e77e6054f6f5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41915 } }
W20260929 03:21:37.329212 9666 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f45e3e7960284dd0b2a6e77e6054f6f5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 03:21:37.329282 9666 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 03:21:37.329219 9653 sys_catalog.cc:455] T 00000000000000000000000000000000 P f45e3e7960284dd0b2a6e77e6054f6f5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f45e3e7960284dd0b2a6e77e6054f6f5. Latest consensus state: current_term: 1 leader_uuid: "f45e3e7960284dd0b2a6e77e6054f6f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f45e3e7960284dd0b2a6e77e6054f6f5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41915 } } }
I20260929 03:21:37.329329 9653 sys_catalog.cc:458] T 00000000000000000000000000000000 P f45e3e7960284dd0b2a6e77e6054f6f5 [sys.catalog]: This master's current role is: LEADER
I20260929 03:21:37.329221 9654 sys_catalog.cc:455] T 00000000000000000000000000000000 P f45e3e7960284dd0b2a6e77e6054f6f5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f45e3e7960284dd0b2a6e77e6054f6f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f45e3e7960284dd0b2a6e77e6054f6f5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41915 } } }
I20260929 03:21:37.329538 9654 sys_catalog.cc:458] T 00000000000000000000000000000000 P f45e3e7960284dd0b2a6e77e6054f6f5 [sys.catalog]: This master's current role is: LEADER
I20260929 03:21:37.329521 9669 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:21:37.329734 9669 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:21:37.331166 9669 catalog_manager.cc:1347] Generated new cluster ID: 77a54b29c34240d8be7fce54c90db61a
I20260929 03:21:37.331216 9669 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 29 03:21:37 dist-test-slave-c9qh krb5kdc[9566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652097, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 03:21:37.336163 9651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:37.324709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.229:50107 (local address 127.0.0.1:41915)
0929 03:21:37.324926 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0929 03:21:37.324937 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 03:21:37.324972 (+ 35us) server_negotiation.cc:407] Connection header received
0929 03:21:37.325203 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:21:37.325212 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:21:37.325274 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:21:37.325429 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:21:37.326264 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:37.327658 (+ 1394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:37.328854 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:37.329469 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:37.329587 (+ 118us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:21:37.332337 (+ 2750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:21:37.332341 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:21:37.332342 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:21:37.332353 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:21:37.334299 (+ 1946us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:21:37.334741 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:21:37.334744 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:21:37.334744 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:21:37.334890 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:21:37.335129 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:21:37.335130 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:21:37.335131 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:21:37.335458 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:21:37.335633 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0929 03:21:37.335714 (+ 81us) server_negotiation.cc:299] Negotiation successful
0929 03:21:37.335816 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":111,"threads_started":1}
I20260929 03:21:37.337232 21193 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 03:21:37.338310 9669 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:21:37.339056 9669 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:21:37.344039 9669 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f45e3e7960284dd0b2a6e77e6054f6f5: Generated new TSK 0
I20260929 03:21:37.344185 9669 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 03:21:37.346396 9651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:37.338660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.229:34695 (local address 127.0.0.1:41915)
0929 03:21:37.338728 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0929 03:21:37.338730 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:21:37.338953 (+ 223us) server_negotiation.cc:407] Connection header received
0929 03:21:37.339012 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:21:37.339013 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:21:37.339052 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:21:37.339105 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:21:37.339737 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:37.340501 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:37.341586 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:37.341889 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:37.341943 (+ 54us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:21:37.343258 (+ 1315us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:21:37.343259 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:21:37.343259 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:21:37.343265 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:21:37.344830 (+ 1565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:21:37.345252 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:21:37.345253 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:21:37.345254 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:21:37.345342 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:21:37.345606 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:21:37.345607 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:21:37.345608 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:21:37.345727 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:21:37.345800 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 03:21:37.346091 (+ 291us) server_negotiation.cc:299] Negotiation successful
0929 03:21:37.346228 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260929 03:21:37.352113 21193 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuoaHbX/build/release/bin/kudu with pid 9579
W20260929 03:21:37.356349 9564 connection.cc:535] client connection to 127.0.0.1:41915 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41915): Connection reset by peer (error 104)
2026-09-29T03:21:37Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1319 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790651852476208-21193-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:21:37 dist-test-slave-c9qh krb5kdc[9678](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:21:37 dist-test-slave-c9qh krb5kdc[9678](info): set up 2 sockets
Sep 29 03:21:37 dist-test-slave-c9qh krb5kdc[9678](info): commencing operation
krb5kdc: starting...
W20260929 03:21:38.468823 21193 mini_kdc.cc:121] Time spent starting KDC: real 1.099s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790651852476208-21193-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790651852476208-21193-0/krb5kdc/test-user.keytab.
Sep 29 03:21:38 dist-test-slave-c9qh krb5kdc[9678](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652098, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:21:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:21:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790651852476208-21193-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790651852476208-21193-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 03:21:38.583124 21193 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuoaHbX/build/release/bin/kudu
/tmp/dist-test-taskuoaHbX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790651852476208-21193-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790651852476208-21193-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790651852476208-21193-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790651852476208-21193-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42489
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790651852476208-21193-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42489
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790651852476208-21193-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790651852476208-21193-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790651852476208-21193-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:21:38.662303 9691 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:21:38.662494 9691 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:21:38.662518 9691 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:21:38.663662 9691 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:21:38.663705 9691 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:21:38.663719 9691 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:21:38.663733 9691 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:21:38.663744 9691 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:21:38.665238 9691 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790651852476208-21193-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790651852476208-21193-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42489
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790651852476208-21193-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42489
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790651852476208-21193-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9691
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790651852476208-21193-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:21:38.665509 9691 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:21:38.665736 9691 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:21:38.670529 9696 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:21:38.670540 9697 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:21:38.670595 9691 server_base.cc:1034] running on GCE node
W20260929 03:21:38.670533 9699 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:21:38.670924 9691 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:21:38.671250 9691 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:21:38.672391 9691 hybrid_clock.cc:648] HybridClock initialized: now 1790652098672381 us; error 28 us; skew 500 ppm
Sep 29 03:21:38 dist-test-slave-c9qh krb5kdc[9678](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652098, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:21:38.685364 9691 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 03:21:38.685853 9691 webserver.cc:460] Webserver started at http://127.0.0.1:39791/ using document root <none> and password file <none>
I20260929 03:21:38.686103 9691 fs_manager.cc:359] Metadata directory not provided
I20260929 03:21:38.686167 9691 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:21:38.686342 9691 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:21:38.687470 9691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790651852476208-21193-0/minicluster-data/master-0/data/instance:
uuid: "79fbd67fff1e49b995237255de487c0a"
format_stamp: "Formatted at 2026-09-29 03:21:38 on dist-test-slave-c9qh"
server_key: "ea5d6ee34a3abcad119d1b0e7130267c"
server_key_iv: "f11b9e7194b85b3e87e1249d58389bfa"
server_key_version: "encryptionkey@0"
I20260929 03:21:38.687815 9691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790651852476208-21193-0/minicluster-data/master-0/wal/instance:
uuid: "79fbd67fff1e49b995237255de487c0a"
format_stamp: "Formatted at 2026-09-29 03:21:38 on dist-test-slave-c9qh"
server_key: "ea5d6ee34a3abcad119d1b0e7130267c"
server_key_iv: "f11b9e7194b85b3e87e1249d58389bfa"
server_key_version: "encryptionkey@0"
I20260929 03:21:38.689355 9691 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 03:21:38.690143 9706 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:21:38.690379 9691 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260929 03:21:38.690469 9691 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790651852476208-21193-0/minicluster-data/master-0/data,/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790651852476208-21193-0/minicluster-data/master-0/wal
uuid: "79fbd67fff1e49b995237255de487c0a"
format_stamp: "Formatted at 2026-09-29 03:21:38 on dist-test-slave-c9qh"
server_key: "ea5d6ee34a3abcad119d1b0e7130267c"
server_key_iv: "f11b9e7194b85b3e87e1249d58389bfa"
server_key_version: "encryptionkey@0"
I20260929 03:21:38.690542 9691 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790651852476208-21193-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790651852476208-21193-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790651852476208-21193-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:21:38.698277 9691 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:21:38.700119 9691 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:21:38.700261 9691 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:21:38.704104 9691 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42489
I20260929 03:21:38.704140 9758 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42489 every 8 connection(s)
I20260929 03:21:38.704475 9691 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790651852476208-21193-0/minicluster-data/master-0/data/info.pb
I20260929 03:21:38.705358 9759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:21:38.707458 21193 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuoaHbX/build/release/bin/kudu as pid 9691
I20260929 03:21:38.707545 21193 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790651852476208-21193-0/minicluster-data/master-0/wal/instance
I20260929 03:21:38.707729 21193 external_mini_cluster.cc:1428] Setting key c07744c9601096873bb731245b1a0c56
I20260929 03:21:38.707867 9759 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79fbd67fff1e49b995237255de487c0a: Bootstrap starting.
I20260929 03:21:38.708576 9759 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 79fbd67fff1e49b995237255de487c0a: Neither blocks nor log segments found. Creating new log.
I20260929 03:21:38.708925 9759 log.cc:824] T 00000000000000000000000000000000 P 79fbd67fff1e49b995237255de487c0a: Log is configured to *not* fsync() on all Append() calls
I20260929 03:21:38.709743 9759 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79fbd67fff1e49b995237255de487c0a: No bootstrap required, opened a new log
I20260929 03:21:38.712110 9759 raft_consensus.cc:348] T 00000000000000000000000000000000 P 79fbd67fff1e49b995237255de487c0a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79fbd67fff1e49b995237255de487c0a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42489 } }
I20260929 03:21:38.712306 9759 raft_consensus.cc:374] T 00000000000000000000000000000000 P 79fbd67fff1e49b995237255de487c0a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:21:38.712360 9759 raft_consensus.cc:729] T 00000000000000000000000000000000 P 79fbd67fff1e49b995237255de487c0a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79fbd67fff1e49b995237255de487c0a, State: Initialized, Role: FOLLOWER
I20260929 03:21:38.712541 9759 consensus_queue.cc:260] T 00000000000000000000000000000000 P 79fbd67fff1e49b995237255de487c0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79fbd67fff1e49b995237255de487c0a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42489 } }
I20260929 03:21:38.712687 9759 raft_consensus.cc:388] T 00000000000000000000000000000000 P 79fbd67fff1e49b995237255de487c0a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:21:38.712749 9759 raft_consensus.cc:482] T 00000000000000000000000000000000 P 79fbd67fff1e49b995237255de487c0a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:21:38.712807 9759 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 79fbd67fff1e49b995237255de487c0a [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:21:38.713426 9759 raft_consensus.cc:504] T 00000000000000000000000000000000 P 79fbd67fff1e49b995237255de487c0a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79fbd67fff1e49b995237255de487c0a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42489 } }
I20260929 03:21:38.713559 9762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:38.708290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.229:59897 (local address 127.0.0.1:42489)
0929 03:21:38.708485 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0929 03:21:38.708493 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 03:21:38.708516 (+ 23us) server_negotiation.cc:407] Connection header received
0929 03:21:38.709009 (+ 493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:21:38.709016 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:21:38.709100 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:21:38.709258 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:21:38.709559 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:38.710434 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:38.711671 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:38.712054 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:38.712135 (+ 81us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:21:38.712929 (+ 794us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:21:38.712933 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:21:38.712933 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:21:38.713101 (+ 168us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:21:38.713130 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:21:38.713141 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:21:38.713250 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0929 03:21:38.713344 (+ 94us) server_negotiation.cc:299] Negotiation successful
0929 03:21:38.713371 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":84,"threads_started":1}
I20260929 03:21:38.713596 9759 leader_election.cc:302] T 00000000000000000000000000000000 P 79fbd67fff1e49b995237255de487c0a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 79fbd67fff1e49b995237255de487c0a; no voters:
I20260929 03:21:38.714005 9759 leader_election.cc:288] T 00000000000000000000000000000000 P 79fbd67fff1e49b995237255de487c0a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:21:38.714069 9764 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 79fbd67fff1e49b995237255de487c0a [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:21:38.714236 9764 raft_consensus.cc:686] T 00000000000000000000000000000000 P 79fbd67fff1e49b995237255de487c0a [term 1 LEADER]: Becoming Leader. State: Replica: 79fbd67fff1e49b995237255de487c0a, State: Running, Role: LEADER
I20260929 03:21:38.714306 9759 sys_catalog.cc:564] T 00000000000000000000000000000000 P 79fbd67fff1e49b995237255de487c0a [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:21:38.714376 9764 consensus_queue.cc:237] T 00000000000000000000000000000000 P 79fbd67fff1e49b995237255de487c0a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79fbd67fff1e49b995237255de487c0a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42489 } }
I20260929 03:21:38.714592 21193 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 03:21:38.714691 9764 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79fbd67fff1e49b995237255de487c0a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 79fbd67fff1e49b995237255de487c0a. Latest consensus state: current_term: 1 leader_uuid: "79fbd67fff1e49b995237255de487c0a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79fbd67fff1e49b995237255de487c0a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42489 } } }
I20260929 03:21:38.714773 9764 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79fbd67fff1e49b995237255de487c0a [sys.catalog]: This master's current role is: LEADER
I20260929 03:21:38.714716 9766 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79fbd67fff1e49b995237255de487c0a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "79fbd67fff1e49b995237255de487c0a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79fbd67fff1e49b995237255de487c0a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42489 } } }
I20260929 03:21:38.714916 9766 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79fbd67fff1e49b995237255de487c0a [sys.catalog]: This master's current role is: LEADER
I20260929 03:21:38.715569 9769 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:21:38.716049 9769 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:21:38.717988 9769 catalog_manager.cc:1347] Generated new cluster ID: 481fb183bc1c41e4b8ba84c74475ee51
I20260929 03:21:38.718051 9769 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:21:38.720184 9762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:38.716452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.229:53425 (local address 127.0.0.1:42489)
0929 03:21:38.716488 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0929 03:21:38.716490 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:21:38.716592 (+ 102us) server_negotiation.cc:407] Connection header received
0929 03:21:38.716692 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:21:38.716693 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:21:38.716702 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:21:38.716758 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:21:38.716904 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:38.717684 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:38.718867 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:38.719118 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:38.719201 (+ 83us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:21:38.719920 (+ 719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:21:38.719922 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:21:38.719922 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:21:38.719927 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:21:38.719942 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:21:38.719946 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:21:38.720006 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0929 03:21:38.720073 (+ 67us) server_negotiation.cc:299] Negotiation successful
0929 03:21:38.720083 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260929 03:21:38.739030 9769 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:21:38.739713 9769 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:21:38.745426 9769 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 79fbd67fff1e49b995237255de487c0a: Generated new TSK 0
I20260929 03:21:38.745589 9769 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 03:21:38.753572 21193 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuoaHbX/build/release/bin/kudu with pid 9691
W20260929 03:21:38.758126 9676 connection.cc:535] client connection to 127.0.0.1:42489 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42489): Connection reset by peer (error 104)
2026-09-29T03:21:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1401 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790651852476208-21193-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:21:38 dist-test-slave-c9qh krb5kdc[9790](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:21:38 dist-test-slave-c9qh krb5kdc[9790](info): set up 2 sockets
Sep 29 03:21:38 dist-test-slave-c9qh krb5kdc[9790](info): commencing operation
krb5kdc: starting...
W20260929 03:21:39.794279 21193 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790651852476208-21193-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790651852476208-21193-0/krb5kdc/test-user.keytab.
Sep 29 03:21:39 dist-test-slave-c9qh krb5kdc[9790](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652099, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:21:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:21:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790651852476208-21193-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790651852476208-21193-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 03:21:39.912617 21193 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuoaHbX/build/release/bin/kudu
/tmp/dist-test-taskuoaHbX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790651852476208-21193-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790651852476208-21193-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790651852476208-21193-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790651852476208-21193-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43105
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790651852476208-21193-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43105
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790651852476208-21193-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790651852476208-21193-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790651852476208-21193-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:21:39.990469 9803 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:21:39.990686 9803 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:21:39.990723 9803 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:21:39.991864 9803 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:21:39.991950 9803 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:21:39.991976 9803 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:21:39.992003 9803 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:21:39.992027 9803 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:21:39.993536 9803 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790651852476208-21193-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790651852476208-21193-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43105
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790651852476208-21193-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43105
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790651852476208-21193-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9803
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790651852476208-21193-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:21:39.993798 9803 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:21:39.994016 9803 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:21:39.998741 9808 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:21:39.998733 9811 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:21:39.998734 9809 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:21:39.999066 9803 server_base.cc:1034] running on GCE node
I20260929 03:21:39.999300 9803 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:21:39.999572 9803 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:21:40.000715 9803 hybrid_clock.cc:648] HybridClock initialized: now 1790652100000707 us; error 49 us; skew 500 ppm
Sep 29 03:21:40 dist-test-slave-c9qh krb5kdc[9790](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652100, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:21:40.003051 9803 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 03:21:40.003454 9803 webserver.cc:460] Webserver started at http://127.0.0.1:46743/ using document root <none> and password file <none>
I20260929 03:21:40.003675 9803 fs_manager.cc:359] Metadata directory not provided
I20260929 03:21:40.003743 9803 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:21:40.003898 9803 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:21:40.004958 9803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790651852476208-21193-0/minicluster-data/master-0/data/instance:
uuid: "9f27da6847fd4b76b0d976e93ed5f22c"
format_stamp: "Formatted at 2026-09-29 03:21:40 on dist-test-slave-c9qh"
server_key: "f6f40b4faa7bb6ea7bb3b67d57b18e5d"
server_key_iv: "ee636513cf7da2689c69fa2a23340dd9"
server_key_version: "encryptionkey@0"
I20260929 03:21:40.005306 9803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790651852476208-21193-0/minicluster-data/master-0/wal/instance:
uuid: "9f27da6847fd4b76b0d976e93ed5f22c"
format_stamp: "Formatted at 2026-09-29 03:21:40 on dist-test-slave-c9qh"
server_key: "f6f40b4faa7bb6ea7bb3b67d57b18e5d"
server_key_iv: "ee636513cf7da2689c69fa2a23340dd9"
server_key_version: "encryptionkey@0"
I20260929 03:21:40.006745 9803 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 03:21:40.007551 9819 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:21:40.007763 9803 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 03:21:40.007869 9803 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790651852476208-21193-0/minicluster-data/master-0/data,/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790651852476208-21193-0/minicluster-data/master-0/wal
uuid: "9f27da6847fd4b76b0d976e93ed5f22c"
format_stamp: "Formatted at 2026-09-29 03:21:40 on dist-test-slave-c9qh"
server_key: "f6f40b4faa7bb6ea7bb3b67d57b18e5d"
server_key_iv: "ee636513cf7da2689c69fa2a23340dd9"
server_key_version: "encryptionkey@0"
I20260929 03:21:40.007956 9803 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790651852476208-21193-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790651852476208-21193-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790651852476208-21193-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:21:40.011313 9803 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:21:40.013208 9803 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:21:40.013376 9803 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:21:40.017266 9803 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43105
I20260929 03:21:40.017295 9871 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43105 every 8 connection(s)
I20260929 03:21:40.017608 9803 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790651852476208-21193-0/minicluster-data/master-0/data/info.pb
I20260929 03:21:40.018400 9872 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:21:40.020489 9872 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f27da6847fd4b76b0d976e93ed5f22c: Bootstrap starting.
I20260929 03:21:40.021088 9872 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9f27da6847fd4b76b0d976e93ed5f22c: Neither blocks nor log segments found. Creating new log.
I20260929 03:21:40.021329 9872 log.cc:824] T 00000000000000000000000000000000 P 9f27da6847fd4b76b0d976e93ed5f22c: Log is configured to *not* fsync() on all Append() calls
I20260929 03:21:40.022015 9872 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f27da6847fd4b76b0d976e93ed5f22c: No bootstrap required, opened a new log
I20260929 03:21:40.023797 9872 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9f27da6847fd4b76b0d976e93ed5f22c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f27da6847fd4b76b0d976e93ed5f22c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43105 } }
I20260929 03:21:40.023928 9872 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9f27da6847fd4b76b0d976e93ed5f22c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:21:40.023950 9872 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9f27da6847fd4b76b0d976e93ed5f22c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f27da6847fd4b76b0d976e93ed5f22c, State: Initialized, Role: FOLLOWER
I20260929 03:21:40.024047 9872 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9f27da6847fd4b76b0d976e93ed5f22c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f27da6847fd4b76b0d976e93ed5f22c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43105 } }
I20260929 03:21:40.024094 9872 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9f27da6847fd4b76b0d976e93ed5f22c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:21:40.024114 9872 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9f27da6847fd4b76b0d976e93ed5f22c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:21:40.024145 9872 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9f27da6847fd4b76b0d976e93ed5f22c [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:21:40.024612 9872 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9f27da6847fd4b76b0d976e93ed5f22c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f27da6847fd4b76b0d976e93ed5f22c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43105 } }
I20260929 03:21:40.024763 9872 leader_election.cc:302] T 00000000000000000000000000000000 P 9f27da6847fd4b76b0d976e93ed5f22c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9f27da6847fd4b76b0d976e93ed5f22c; no voters:
I20260929 03:21:40.024881 9872 leader_election.cc:288] T 00000000000000000000000000000000 P 9f27da6847fd4b76b0d976e93ed5f22c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:21:40.024930 9875 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9f27da6847fd4b76b0d976e93ed5f22c [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:21:40.025084 9875 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9f27da6847fd4b76b0d976e93ed5f22c [term 1 LEADER]: Becoming Leader. State: Replica: 9f27da6847fd4b76b0d976e93ed5f22c, State: Running, Role: LEADER
I20260929 03:21:40.025142 9872 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9f27da6847fd4b76b0d976e93ed5f22c [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:21:40.025207 9875 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9f27da6847fd4b76b0d976e93ed5f22c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f27da6847fd4b76b0d976e93ed5f22c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43105 } }
I20260929 03:21:40.025602 9876 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f27da6847fd4b76b0d976e93ed5f22c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9f27da6847fd4b76b0d976e93ed5f22c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f27da6847fd4b76b0d976e93ed5f22c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43105 } } }
I20260929 03:21:40.025611 9877 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f27da6847fd4b76b0d976e93ed5f22c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9f27da6847fd4b76b0d976e93ed5f22c. Latest consensus state: current_term: 1 leader_uuid: "9f27da6847fd4b76b0d976e93ed5f22c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f27da6847fd4b76b0d976e93ed5f22c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43105 } } }
I20260929 03:21:40.025769 9876 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f27da6847fd4b76b0d976e93ed5f22c [sys.catalog]: This master's current role is: LEADER
I20260929 03:21:40.025794 9877 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f27da6847fd4b76b0d976e93ed5f22c [sys.catalog]: This master's current role is: LEADER
I20260929 03:21:40.026151 9881 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:21:40.026712 9881 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:21:40.026921 21193 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuoaHbX/build/release/bin/kudu as pid 9803
I20260929 03:21:40.027025 21193 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790651852476208-21193-0/minicluster-data/master-0/wal/instance
I20260929 03:21:40.027215 21193 external_mini_cluster.cc:1428] Setting key dcde216580519cc051999c577d9ba477
I20260929 03:21:40.028242 9881 catalog_manager.cc:1347] Generated new cluster ID: b3529b07b64644ba87c6bafc1a1d1c25
I20260929 03:21:40.028291 9881 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:21:40.034291 9895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:40.027801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.229:34211 (local address 127.0.0.1:43105)
0929 03:21:40.032770 (+ 4969us) server_negotiation.cc:206] Beginning negotiation
0929 03:21:40.032779 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 03:21:40.032816 (+ 37us) server_negotiation.cc:407] Connection header received
0929 03:21:40.032943 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:21:40.032953 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:21:40.032989 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:21:40.033165 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:21:40.033859 (+ 694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:21:40.033862 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:21:40.033863 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:21:40.033867 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:21:40.033902 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:21:40.033916 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:21:40.033988 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0929 03:21:40.034060 (+ 72us) server_negotiation.cc:299] Negotiation successful
0929 03:21:40.034105 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4916,"thread_start_us":90,"threads_started":1}
I20260929 03:21:40.035215 21193 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 03:21:40.038746 9881 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:21:40.038882 9881 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:21:40.039705 9895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:40.036768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.229:47819 (local address 127.0.0.1:43105)
0929 03:21:40.036797 (+ 29us) server_negotiation.cc:206] Beginning negotiation
0929 03:21:40.036798 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 03:21:40.038749 (+ 1951us) server_negotiation.cc:407] Connection header received
0929 03:21:40.038843 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:21:40.038844 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:21:40.038851 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:21:40.038881 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:21:40.039500 (+ 619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:21:40.039501 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:21:40.039502 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:21:40.039506 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:21:40.039520 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:21:40.039524 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:21:40.039578 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0929 03:21:40.039633 (+ 55us) server_negotiation.cc:299] Negotiation successful
0929 03:21:40.039646 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260929 03:21:40.043143 9881 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9f27da6847fd4b76b0d976e93ed5f22c: Generated new TSK 0
I20260929 03:21:40.043289 9881 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 03:21:40.046669 21193 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuoaHbX/build/release/bin/kudu with pid 9803
2026-09-29T03:21:40Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1292 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790651852476208-21193-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:21:40 dist-test-slave-c9qh krb5kdc[9903](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:21:40 dist-test-slave-c9qh krb5kdc[9903](info): set up 2 sockets
Sep 29 03:21:40 dist-test-slave-c9qh krb5kdc[9903](info): commencing operation
krb5kdc: starting...
W20260929 03:21:41.080868 21193 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790651852476208-21193-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790651852476208-21193-0/krb5kdc/test-user.keytab.
Sep 29 03:21:41 dist-test-slave-c9qh krb5kdc[9903](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652101, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:21:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:21:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790651852476208-21193-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790651852476208-21193-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 03:21:41.193210 21193 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuoaHbX/build/release/bin/kudu
/tmp/dist-test-taskuoaHbX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790651852476208-21193-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790651852476208-21193-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790651852476208-21193-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790651852476208-21193-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36041
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790651852476208-21193-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36041
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790651852476208-21193-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790651852476208-21193-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790651852476208-21193-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:21:41.271838 9916 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:21:41.272039 9916 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:21:41.272063 9916 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:21:41.273305 9916 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:21:41.273351 9916 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:21:41.273366 9916 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:21:41.273401 9916 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:21:41.273432 9916 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:21:41.274885 9916 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790651852476208-21193-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790651852476208-21193-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36041
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790651852476208-21193-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36041
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790651852476208-21193-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9916
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790651852476208-21193-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:21:41.275132 9916 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:21:41.275385 9916 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:21:41.280309 9924 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:21:41.280364 9916 server_base.cc:1034] running on GCE node
W20260929 03:21:41.280289 9921 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:21:41.280289 9922 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:21:41.280769 9916 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:21:41.281124 9916 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:21:41.282285 9916 hybrid_clock.cc:648] HybridClock initialized: now 1790652101282256 us; error 47 us; skew 500 ppm
Sep 29 03:21:41 dist-test-slave-c9qh krb5kdc[9903](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652101, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:21:41.284749 9916 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 03:21:41.285167 9916 webserver.cc:460] Webserver started at http://127.0.0.1:35167/ using document root <none> and password file <none>
I20260929 03:21:41.285357 9916 fs_manager.cc:359] Metadata directory not provided
I20260929 03:21:41.285396 9916 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:21:41.285483 9916 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:21:41.286434 9916 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790651852476208-21193-0/minicluster-data/master-0/data/instance:
uuid: "a34566fdd72b41ed9f659f94b5b2ebbe"
format_stamp: "Formatted at 2026-09-29 03:21:41 on dist-test-slave-c9qh"
server_key: "8d2065c239ad6446947ad18a9f05ed14"
server_key_iv: "6d0ec7c7996ff9ac84cfcf4d1a0a29e2"
server_key_version: "encryptionkey@0"
I20260929 03:21:41.286720 9916 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790651852476208-21193-0/minicluster-data/master-0/wal/instance:
uuid: "a34566fdd72b41ed9f659f94b5b2ebbe"
format_stamp: "Formatted at 2026-09-29 03:21:41 on dist-test-slave-c9qh"
server_key: "8d2065c239ad6446947ad18a9f05ed14"
server_key_iv: "6d0ec7c7996ff9ac84cfcf4d1a0a29e2"
server_key_version: "encryptionkey@0"
I20260929 03:21:41.287945 9916 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 03:21:41.288601 9931 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:21:41.288777 9916 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 03:21:41.288848 9916 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790651852476208-21193-0/minicluster-data/master-0/data,/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790651852476208-21193-0/minicluster-data/master-0/wal
uuid: "a34566fdd72b41ed9f659f94b5b2ebbe"
format_stamp: "Formatted at 2026-09-29 03:21:41 on dist-test-slave-c9qh"
server_key: "8d2065c239ad6446947ad18a9f05ed14"
server_key_iv: "6d0ec7c7996ff9ac84cfcf4d1a0a29e2"
server_key_version: "encryptionkey@0"
I20260929 03:21:41.288898 9916 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790651852476208-21193-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790651852476208-21193-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790651852476208-21193-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:21:41.301205 9916 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:21:41.302979 9916 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:21:41.303115 9916 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:21:41.307125 9916 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36041
I20260929 03:21:41.307163 9983 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36041 every 8 connection(s)
I20260929 03:21:41.307480 9916 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790651852476208-21193-0/minicluster-data/master-0/data/info.pb
I20260929 03:21:41.308259 9984 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:21:41.310369 9984 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a34566fdd72b41ed9f659f94b5b2ebbe: Bootstrap starting.
I20260929 03:21:41.310899 9984 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a34566fdd72b41ed9f659f94b5b2ebbe: Neither blocks nor log segments found. Creating new log.
I20260929 03:21:41.311167 9984 log.cc:824] T 00000000000000000000000000000000 P a34566fdd72b41ed9f659f94b5b2ebbe: Log is configured to *not* fsync() on all Append() calls
I20260929 03:21:41.311853 9984 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a34566fdd72b41ed9f659f94b5b2ebbe: No bootstrap required, opened a new log
I20260929 03:21:41.313488 9984 raft_consensus.cc:348] T 00000000000000000000000000000000 P a34566fdd72b41ed9f659f94b5b2ebbe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a34566fdd72b41ed9f659f94b5b2ebbe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36041 } }
I20260929 03:21:41.313643 9984 raft_consensus.cc:374] T 00000000000000000000000000000000 P a34566fdd72b41ed9f659f94b5b2ebbe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:21:41.313687 9984 raft_consensus.cc:729] T 00000000000000000000000000000000 P a34566fdd72b41ed9f659f94b5b2ebbe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a34566fdd72b41ed9f659f94b5b2ebbe, State: Initialized, Role: FOLLOWER
I20260929 03:21:41.313868 9984 consensus_queue.cc:260] T 00000000000000000000000000000000 P a34566fdd72b41ed9f659f94b5b2ebbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a34566fdd72b41ed9f659f94b5b2ebbe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36041 } }
I20260929 03:21:41.313974 9984 raft_consensus.cc:388] T 00000000000000000000000000000000 P a34566fdd72b41ed9f659f94b5b2ebbe [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:21:41.314018 9984 raft_consensus.cc:482] T 00000000000000000000000000000000 P a34566fdd72b41ed9f659f94b5b2ebbe [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:21:41.314069 9984 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a34566fdd72b41ed9f659f94b5b2ebbe [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:21:41.314565 9984 raft_consensus.cc:504] T 00000000000000000000000000000000 P a34566fdd72b41ed9f659f94b5b2ebbe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a34566fdd72b41ed9f659f94b5b2ebbe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36041 } }
I20260929 03:21:41.314680 9984 leader_election.cc:302] T 00000000000000000000000000000000 P a34566fdd72b41ed9f659f94b5b2ebbe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a34566fdd72b41ed9f659f94b5b2ebbe; no voters:
I20260929 03:21:41.314846 9984 leader_election.cc:288] T 00000000000000000000000000000000 P a34566fdd72b41ed9f659f94b5b2ebbe [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:21:41.314867 9987 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a34566fdd72b41ed9f659f94b5b2ebbe [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:21:41.315042 9987 raft_consensus.cc:686] T 00000000000000000000000000000000 P a34566fdd72b41ed9f659f94b5b2ebbe [term 1 LEADER]: Becoming Leader. State: Replica: a34566fdd72b41ed9f659f94b5b2ebbe, State: Running, Role: LEADER
I20260929 03:21:41.315086 9984 sys_catalog.cc:564] T 00000000000000000000000000000000 P a34566fdd72b41ed9f659f94b5b2ebbe [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:21:41.315131 9987 consensus_queue.cc:237] T 00000000000000000000000000000000 P a34566fdd72b41ed9f659f94b5b2ebbe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a34566fdd72b41ed9f659f94b5b2ebbe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36041 } }
I20260929 03:21:41.315702 9989 sys_catalog.cc:455] T 00000000000000000000000000000000 P a34566fdd72b41ed9f659f94b5b2ebbe [sys.catalog]: SysCatalogTable state changed. Reason: New leader a34566fdd72b41ed9f659f94b5b2ebbe. Latest consensus state: current_term: 1 leader_uuid: "a34566fdd72b41ed9f659f94b5b2ebbe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a34566fdd72b41ed9f659f94b5b2ebbe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36041 } } }
I20260929 03:21:41.315795 9989 sys_catalog.cc:458] T 00000000000000000000000000000000 P a34566fdd72b41ed9f659f94b5b2ebbe [sys.catalog]: This master's current role is: LEADER
I20260929 03:21:41.315945 9988 sys_catalog.cc:455] T 00000000000000000000000000000000 P a34566fdd72b41ed9f659f94b5b2ebbe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a34566fdd72b41ed9f659f94b5b2ebbe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a34566fdd72b41ed9f659f94b5b2ebbe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36041 } } }
I20260929 03:21:41.316002 9996 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:21:41.316012 9988 sys_catalog.cc:458] T 00000000000000000000000000000000 P a34566fdd72b41ed9f659f94b5b2ebbe [sys.catalog]: This master's current role is: LEADER
I20260929 03:21:41.316540 9996 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:21:41.317384 21193 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuoaHbX/build/release/bin/kudu as pid 9916
I20260929 03:21:41.317483 21193 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790651852476208-21193-0/minicluster-data/master-0/wal/instance
I20260929 03:21:41.317662 21193 external_mini_cluster.cc:1428] Setting key a70a4fe813874e6cbe50fba0b52fc73e
I20260929 03:21:41.321035 9996 catalog_manager.cc:1347] Generated new cluster ID: 1ed278e2b3554244a4e20e111eaa9701
I20260929 03:21:41.321105 9996 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 29 03:21:41 dist-test-slave-c9qh krb5kdc[9903](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652101, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 03:21:41.331956 10007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:41.318113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.229:34847 (local address 127.0.0.1:36041)
0929 03:21:41.318271 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0929 03:21:41.318278 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:21:41.321195 (+ 2917us) server_negotiation.cc:407] Connection header received
0929 03:21:41.321391 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:21:41.321398 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:21:41.321470 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:21:41.321600 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:21:41.322332 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:41.323698 (+ 1366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:41.324806 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:41.325420 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:41.325515 (+ 95us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:21:41.328286 (+ 2771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:21:41.328290 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:21:41.328290 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:21:41.328302 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:21:41.330259 (+ 1957us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:21:41.330702 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:21:41.330705 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:21:41.330706 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:21:41.330811 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:21:41.331046 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:21:41.331047 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:21:41.331047 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:21:41.331343 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:21:41.331516 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0929 03:21:41.331625 (+ 109us) server_negotiation.cc:299] Negotiation successful
0929 03:21:41.331725 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":67,"threads_started":1}
I20260929 03:21:41.332875 21193 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 03:21:41.332911 9996 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:21:41.333559 9996 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:21:41.336149 9996 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a34566fdd72b41ed9f659f94b5b2ebbe: Generated new TSK 0
I20260929 03:21:41.336304 9996 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 03:21:41.341177 10007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:41.334343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.229:43731 (local address 127.0.0.1:36041)
0929 03:21:41.334376 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0929 03:21:41.334378 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:21:41.334498 (+ 120us) server_negotiation.cc:407] Connection header received
0929 03:21:41.334551 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:21:41.334552 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:21:41.334561 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:21:41.334620 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:21:41.335192 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:41.335927 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:41.337001 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:41.337192 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:41.337255 (+ 63us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:21:41.338541 (+ 1286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:21:41.338542 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:21:41.338542 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:21:41.338548 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:21:41.340012 (+ 1464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:21:41.340402 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:21:41.340402 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:21:41.340403 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:21:41.340450 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:21:41.340737 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:21:41.340738 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:21:41.340738 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:21:41.340829 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:21:41.340907 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0929 03:21:41.341054 (+ 147us) server_negotiation.cc:299] Negotiation successful
0929 03:21:41.341101 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":10,"server-negotiator.queue_time_us":17}
I20260929 03:21:41.343166 21193 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuoaHbX/build/release/bin/kudu with pid 9916
W20260929 03:21:41.347064 9901 connection.cc:535] client connection to 127.0.0.1:36041 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36041): Connection reset by peer (error 104)
2026-09-29T03:21:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1295 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790651852476208-21193-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:21:41 dist-test-slave-c9qh krb5kdc[10015](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:21:41 dist-test-slave-c9qh krb5kdc[10015](info): set up 2 sockets
Sep 29 03:21:41 dist-test-slave-c9qh krb5kdc[10015](info): commencing operation
krb5kdc: starting...
W20260929 03:21:42.361040 21193 mini_kdc.cc:121] Time spent starting KDC: real 1.000s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790651852476208-21193-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790651852476208-21193-0/krb5kdc/test-user.keytab.
Sep 29 03:21:42 dist-test-slave-c9qh krb5kdc[10015](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652102, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:21:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:21:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790651852476208-21193-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790651852476208-21193-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 03:21:42.481357 21193 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuoaHbX/build/release/bin/kudu
/tmp/dist-test-taskuoaHbX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790651852476208-21193-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790651852476208-21193-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790651852476208-21193-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790651852476208-21193-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44151
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790651852476208-21193-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44151
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790651852476208-21193-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790651852476208-21193-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790651852476208-21193-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:21:42.565459 10028 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:21:42.565707 10028 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:21:42.565747 10028 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:21:42.566890 10028 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:21:42.566964 10028 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:21:42.566988 10028 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:21:42.567009 10028 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:21:42.567036 10028 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:21:42.568562 10028 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790651852476208-21193-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790651852476208-21193-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44151
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790651852476208-21193-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44151
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790651852476208-21193-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10028
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790651852476208-21193-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:21:42.568866 10028 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:21:42.569097 10028 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:21:42.573937 10034 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:21:42.573945 10036 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:21:42.573948 10033 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:21:42.574163 10028 server_base.cc:1034] running on GCE node
I20260929 03:21:42.574422 10028 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:21:42.574777 10028 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:21:42.575940 10028 hybrid_clock.cc:648] HybridClock initialized: now 1790652102575928 us; error 25 us; skew 500 ppm
Sep 29 03:21:42 dist-test-slave-c9qh krb5kdc[10015](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652102, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:21:42.578444 10028 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 03:21:42.578915 10028 webserver.cc:460] Webserver started at http://127.0.0.1:41867/ using document root <none> and password file <none>
I20260929 03:21:42.579161 10028 fs_manager.cc:359] Metadata directory not provided
I20260929 03:21:42.579231 10028 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:21:42.579380 10028 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:21:42.580454 10028 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790651852476208-21193-0/minicluster-data/master-0/data/instance:
uuid: "4d82385332b14d1ebc11e1a03f3609cc"
format_stamp: "Formatted at 2026-09-29 03:21:42 on dist-test-slave-c9qh"
server_key: "e32e97208fa0e7149705f8d333213ac5"
server_key_iv: "9b3ccb10bee92f55a5ae3b0acacb8c39"
server_key_version: "encryptionkey@0"
I20260929 03:21:42.580853 10028 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790651852476208-21193-0/minicluster-data/master-0/wal/instance:
uuid: "4d82385332b14d1ebc11e1a03f3609cc"
format_stamp: "Formatted at 2026-09-29 03:21:42 on dist-test-slave-c9qh"
server_key: "e32e97208fa0e7149705f8d333213ac5"
server_key_iv: "9b3ccb10bee92f55a5ae3b0acacb8c39"
server_key_version: "encryptionkey@0"
I20260929 03:21:42.582175 10028 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260929 03:21:42.582881 10043 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:21:42.583055 10028 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 03:21:42.583161 10028 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790651852476208-21193-0/minicluster-data/master-0/data,/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790651852476208-21193-0/minicluster-data/master-0/wal
uuid: "4d82385332b14d1ebc11e1a03f3609cc"
format_stamp: "Formatted at 2026-09-29 03:21:42 on dist-test-slave-c9qh"
server_key: "e32e97208fa0e7149705f8d333213ac5"
server_key_iv: "9b3ccb10bee92f55a5ae3b0acacb8c39"
server_key_version: "encryptionkey@0"
I20260929 03:21:42.583236 10028 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790651852476208-21193-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790651852476208-21193-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790651852476208-21193-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:21:42.615067 10028 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:21:42.617606 10028 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:21:42.617789 10028 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:21:42.622044 10028 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44151
I20260929 03:21:42.622084 10095 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44151 every 8 connection(s)
I20260929 03:21:42.622453 10028 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790651852476208-21193-0/minicluster-data/master-0/data/info.pb
I20260929 03:21:42.623311 10096 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:21:42.625547 10096 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d82385332b14d1ebc11e1a03f3609cc: Bootstrap starting.
I20260929 03:21:42.626072 10096 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4d82385332b14d1ebc11e1a03f3609cc: Neither blocks nor log segments found. Creating new log.
I20260929 03:21:42.626297 10096 log.cc:824] T 00000000000000000000000000000000 P 4d82385332b14d1ebc11e1a03f3609cc: Log is configured to *not* fsync() on all Append() calls
I20260929 03:21:42.626602 21193 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuoaHbX/build/release/bin/kudu as pid 10028
I20260929 03:21:42.626719 21193 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790651852476208-21193-0/minicluster-data/master-0/wal/instance
I20260929 03:21:42.626935 21193 external_mini_cluster.cc:1428] Setting key c904bd0aa58acd3ebd2fd2f9190b10ef
I20260929 03:21:42.627161 10096 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d82385332b14d1ebc11e1a03f3609cc: No bootstrap required, opened a new log
I20260929 03:21:42.630098 10096 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4d82385332b14d1ebc11e1a03f3609cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d82385332b14d1ebc11e1a03f3609cc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44151 } }
I20260929 03:21:42.630263 10096 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4d82385332b14d1ebc11e1a03f3609cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:21:42.630295 10096 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4d82385332b14d1ebc11e1a03f3609cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d82385332b14d1ebc11e1a03f3609cc, State: Initialized, Role: FOLLOWER
I20260929 03:21:42.630424 10096 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4d82385332b14d1ebc11e1a03f3609cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d82385332b14d1ebc11e1a03f3609cc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44151 } }
I20260929 03:21:42.630517 10096 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4d82385332b14d1ebc11e1a03f3609cc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:21:42.630573 10096 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4d82385332b14d1ebc11e1a03f3609cc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:21:42.630622 10096 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4d82385332b14d1ebc11e1a03f3609cc [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:21:42.631217 10096 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4d82385332b14d1ebc11e1a03f3609cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d82385332b14d1ebc11e1a03f3609cc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44151 } }
I20260929 03:21:42.631315 10096 leader_election.cc:302] T 00000000000000000000000000000000 P 4d82385332b14d1ebc11e1a03f3609cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4d82385332b14d1ebc11e1a03f3609cc; no voters:
I20260929 03:21:42.631479 10096 leader_election.cc:288] T 00000000000000000000000000000000 P 4d82385332b14d1ebc11e1a03f3609cc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:21:42.631505 10101 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4d82385332b14d1ebc11e1a03f3609cc [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:21:42.631665 10101 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4d82385332b14d1ebc11e1a03f3609cc [term 1 LEADER]: Becoming Leader. State: Replica: 4d82385332b14d1ebc11e1a03f3609cc, State: Running, Role: LEADER
I20260929 03:21:42.631723 10096 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4d82385332b14d1ebc11e1a03f3609cc [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:21:42.631788 10101 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4d82385332b14d1ebc11e1a03f3609cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d82385332b14d1ebc11e1a03f3609cc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44151 } }
I20260929 03:21:42.632131 10101 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d82385332b14d1ebc11e1a03f3609cc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4d82385332b14d1ebc11e1a03f3609cc. Latest consensus state: current_term: 1 leader_uuid: "4d82385332b14d1ebc11e1a03f3609cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d82385332b14d1ebc11e1a03f3609cc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44151 } } }
I20260929 03:21:42.632141 10103 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d82385332b14d1ebc11e1a03f3609cc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4d82385332b14d1ebc11e1a03f3609cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d82385332b14d1ebc11e1a03f3609cc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44151 } } }
I20260929 03:21:42.632282 10103 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d82385332b14d1ebc11e1a03f3609cc [sys.catalog]: This master's current role is: LEADER
I20260929 03:21:42.632478 10101 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d82385332b14d1ebc11e1a03f3609cc [sys.catalog]: This master's current role is: LEADER
I20260929 03:21:42.632661 10108 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:21:42.633190 10108 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:21:42.633884 10100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:42.627468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.229:37111 (local address 127.0.0.1:44151)
0929 03:21:42.627661 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0929 03:21:42.627672 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 03:21:42.627707 (+ 35us) server_negotiation.cc:407] Connection header received
0929 03:21:42.628525 (+ 818us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:21:42.628535 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:21:42.628604 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:21:42.628779 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:21:42.628935 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:42.630345 (+ 1410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:42.631203 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:42.631850 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:42.632872 (+ 1022us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:21:42.633053 (+ 181us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:21:42.633056 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:21:42.633056 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:21:42.633293 (+ 237us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:21:42.633338 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:21:42.633355 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:21:42.633630 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0929 03:21:42.633680 (+ 50us) server_negotiation.cc:299] Negotiation successful
0929 03:21:42.633708 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":86,"threads_started":1}
I20260929 03:21:42.634900 10108 catalog_manager.cc:1347] Generated new cluster ID: 5d452105bda74171a5d53a5814732035
I20260929 03:21:42.634954 10108 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:21:42.634967 21193 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 03:21:42.640245 10100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:42.636409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.229:49805 (local address 127.0.0.1:44151)
0929 03:21:42.636453 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0929 03:21:42.636455 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:21:42.636905 (+ 450us) server_negotiation.cc:407] Connection header received
0929 03:21:42.636985 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:21:42.636986 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:21:42.636995 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:21:42.637041 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:21:42.637182 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:42.637875 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:42.638875 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:42.639187 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:42.639244 (+ 57us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:21:42.639972 (+ 728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:21:42.639973 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:21:42.639974 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:21:42.639978 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:21:42.639991 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:21:42.639995 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:21:42.640049 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0929 03:21:42.640150 (+ 101us) server_negotiation.cc:299] Negotiation successful
0929 03:21:42.640172 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":10,"server-negotiator.queue_time_us":16}
I20260929 03:21:42.645934 10108 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:21:42.646648 10108 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:21:42.650132 10108 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4d82385332b14d1ebc11e1a03f3609cc: Generated new TSK 0
I20260929 03:21:42.650302 10108 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 03:21:42.657369 21193 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuoaHbX/build/release/bin/kudu with pid 10028
W20260929 03:21:42.661896 10013 connection.cc:535] client connection to 127.0.0.1:44151 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44151): Connection reset by peer (error 104)
2026-09-29T03:21:42Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1314 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790651852476208-21193-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:21:42 dist-test-slave-c9qh krb5kdc[10127](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:21:42 dist-test-slave-c9qh krb5kdc[10127](info): set up 2 sockets
Sep 29 03:21:42 dist-test-slave-c9qh krb5kdc[10127](info): commencing operation
krb5kdc: starting...
W20260929 03:21:43.680719 21193 mini_kdc.cc:121] Time spent starting KDC: real 1.005s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790651852476208-21193-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790651852476208-21193-0/krb5kdc/test-user.keytab.
Sep 29 03:21:43 dist-test-slave-c9qh krb5kdc[10127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652103, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:21:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:21:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790651852476208-21193-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790651852476208-21193-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 03:21:43.797657 21193 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuoaHbX/build/release/bin/kudu
/tmp/dist-test-taskuoaHbX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790651852476208-21193-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790651852476208-21193-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790651852476208-21193-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790651852476208-21193-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46441
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790651852476208-21193-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46441
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790651852476208-21193-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790651852476208-21193-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790651852476208-21193-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:21:43.877089 10140 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:21:43.877285 10140 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:21:43.877314 10140 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:21:43.878458 10140 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:21:43.878499 10140 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:21:43.878512 10140 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:21:43.878525 10140 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:21:43.878580 10140 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:21:43.879992 10140 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790651852476208-21193-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790651852476208-21193-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46441
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790651852476208-21193-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46441
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790651852476208-21193-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10140
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790651852476208-21193-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:21:43.880239 10140 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:21:43.880472 10140 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:21:43.885296 10148 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:21:43.885308 10145 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:21:43.885282 10146 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:21:43.885602 10140 server_base.cc:1034] running on GCE node
I20260929 03:21:43.885835 10140 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:21:43.886125 10140 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:21:43.887297 10140 hybrid_clock.cc:648] HybridClock initialized: now 1790652103887293 us; error 37 us; skew 500 ppm
Sep 29 03:21:43 dist-test-slave-c9qh krb5kdc[10127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652103, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:21:43.895399 10140 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 03:21:43.895809 10140 webserver.cc:460] Webserver started at http://127.0.0.1:40069/ using document root <none> and password file <none>
I20260929 03:21:43.895996 10140 fs_manager.cc:359] Metadata directory not provided
I20260929 03:21:43.896037 10140 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:21:43.896130 10140 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:21:43.897226 10140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790651852476208-21193-0/minicluster-data/master-0/data/instance:
uuid: "826f8c5f00154e17ac876679f3d43bca"
format_stamp: "Formatted at 2026-09-29 03:21:43 on dist-test-slave-c9qh"
server_key: "b993512ee335cfea82afe5bf7ead6f62"
server_key_iv: "2d31c8069e98467e616acb5e4a49900c"
server_key_version: "encryptionkey@0"
I20260929 03:21:43.897538 10140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790651852476208-21193-0/minicluster-data/master-0/wal/instance:
uuid: "826f8c5f00154e17ac876679f3d43bca"
format_stamp: "Formatted at 2026-09-29 03:21:43 on dist-test-slave-c9qh"
server_key: "b993512ee335cfea82afe5bf7ead6f62"
server_key_iv: "2d31c8069e98467e616acb5e4a49900c"
server_key_version: "encryptionkey@0"
I20260929 03:21:43.898820 10140 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 03:21:43.899515 10155 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:21:43.899698 10140 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 03:21:43.899789 10140 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790651852476208-21193-0/minicluster-data/master-0/data,/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790651852476208-21193-0/minicluster-data/master-0/wal
uuid: "826f8c5f00154e17ac876679f3d43bca"
format_stamp: "Formatted at 2026-09-29 03:21:43 on dist-test-slave-c9qh"
server_key: "b993512ee335cfea82afe5bf7ead6f62"
server_key_iv: "2d31c8069e98467e616acb5e4a49900c"
server_key_version: "encryptionkey@0"
I20260929 03:21:43.899847 10140 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790651852476208-21193-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790651852476208-21193-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790651852476208-21193-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:21:43.903098 10140 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:21:43.905404 10140 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:21:43.905512 10140 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:21:43.909292 10140 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46441
I20260929 03:21:43.909309 10207 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46441 every 8 connection(s)
I20260929 03:21:43.909688 10140 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790651852476208-21193-0/minicluster-data/master-0/data/info.pb
I20260929 03:21:43.910678 10208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:21:43.912142 21193 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuoaHbX/build/release/bin/kudu as pid 10140
I20260929 03:21:43.912258 21193 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790651852476208-21193-0/minicluster-data/master-0/wal/instance
I20260929 03:21:43.912449 21193 external_mini_cluster.cc:1428] Setting key 93b97b04c91fe5c0a885cf9554874548
I20260929 03:21:43.913416 10208 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 826f8c5f00154e17ac876679f3d43bca: Bootstrap starting.
I20260929 03:21:43.914105 10208 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 826f8c5f00154e17ac876679f3d43bca: Neither blocks nor log segments found. Creating new log.
I20260929 03:21:43.914434 10208 log.cc:824] T 00000000000000000000000000000000 P 826f8c5f00154e17ac876679f3d43bca: Log is configured to *not* fsync() on all Append() calls
I20260929 03:21:43.915232 10211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:43.912991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.229:58455 (local address 127.0.0.1:46441)
0929 03:21:43.913185 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0929 03:21:43.913195 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 03:21:43.913227 (+ 32us) server_negotiation.cc:407] Connection header received
0929 03:21:43.913810 (+ 583us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:21:43.913821 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:21:43.913855 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:21:43.914000 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:21:43.914684 (+ 684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:21:43.914690 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:21:43.914694 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:21:43.914698 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:21:43.914731 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:21:43.914741 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:21:43.914811 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0929 03:21:43.914894 (+ 83us) server_negotiation.cc:299] Negotiation successful
0929 03:21:43.915054 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":80,"threads_started":1}
I20260929 03:21:43.915246 10208 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 826f8c5f00154e17ac876679f3d43bca: No bootstrap required, opened a new log
I20260929 03:21:43.916165 21193 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 03:21:43.917513 10208 raft_consensus.cc:348] T 00000000000000000000000000000000 P 826f8c5f00154e17ac876679f3d43bca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "826f8c5f00154e17ac876679f3d43bca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46441 } }
I20260929 03:21:43.917650 10208 raft_consensus.cc:374] T 00000000000000000000000000000000 P 826f8c5f00154e17ac876679f3d43bca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:21:43.917672 10208 raft_consensus.cc:729] T 00000000000000000000000000000000 P 826f8c5f00154e17ac876679f3d43bca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 826f8c5f00154e17ac876679f3d43bca, State: Initialized, Role: FOLLOWER
I20260929 03:21:43.917815 10208 consensus_queue.cc:260] T 00000000000000000000000000000000 P 826f8c5f00154e17ac876679f3d43bca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "826f8c5f00154e17ac876679f3d43bca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46441 } }
I20260929 03:21:43.917934 10208 raft_consensus.cc:388] T 00000000000000000000000000000000 P 826f8c5f00154e17ac876679f3d43bca [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:21:43.917985 10208 raft_consensus.cc:482] T 00000000000000000000000000000000 P 826f8c5f00154e17ac876679f3d43bca [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:21:43.918042 10208 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 826f8c5f00154e17ac876679f3d43bca [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:21:43.918658 10208 raft_consensus.cc:504] T 00000000000000000000000000000000 P 826f8c5f00154e17ac876679f3d43bca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "826f8c5f00154e17ac876679f3d43bca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46441 } }
I20260929 03:21:43.918783 10208 leader_election.cc:302] T 00000000000000000000000000000000 P 826f8c5f00154e17ac876679f3d43bca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 826f8c5f00154e17ac876679f3d43bca; no voters:
I20260929 03:21:43.918955 10208 leader_election.cc:288] T 00000000000000000000000000000000 P 826f8c5f00154e17ac876679f3d43bca [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:21:43.919083 10211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:43.917977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.229:53057 (local address 127.0.0.1:46441)
0929 03:21:43.918014 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0929 03:21:43.918016 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:21:43.918131 (+ 115us) server_negotiation.cc:407] Connection header received
0929 03:21:43.918199 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:21:43.918200 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:21:43.918206 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:21:43.918269 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:21:43.918847 (+ 578us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:21:43.918848 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:21:43.918848 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:21:43.918853 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:21:43.918867 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:21:43.918872 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:21:43.918946 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0929 03:21:43.918993 (+ 47us) server_negotiation.cc:299] Negotiation successful
0929 03:21:43.919006 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260929 03:21:43.919126 10219 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 826f8c5f00154e17ac876679f3d43bca [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:21:43.919219 10208 sys_catalog.cc:564] T 00000000000000000000000000000000 P 826f8c5f00154e17ac876679f3d43bca [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:21:43.919268 10219 raft_consensus.cc:686] T 00000000000000000000000000000000 P 826f8c5f00154e17ac876679f3d43bca [term 1 LEADER]: Becoming Leader. State: Replica: 826f8c5f00154e17ac876679f3d43bca, State: Running, Role: LEADER
I20260929 03:21:43.919344 10219 consensus_queue.cc:237] T 00000000000000000000000000000000 P 826f8c5f00154e17ac876679f3d43bca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "826f8c5f00154e17ac876679f3d43bca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46441 } }
I20260929 03:21:43.919646 10219 sys_catalog.cc:455] T 00000000000000000000000000000000 P 826f8c5f00154e17ac876679f3d43bca [sys.catalog]: SysCatalogTable state changed. Reason: New leader 826f8c5f00154e17ac876679f3d43bca. Latest consensus state: current_term: 1 leader_uuid: "826f8c5f00154e17ac876679f3d43bca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "826f8c5f00154e17ac876679f3d43bca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46441 } } }
I20260929 03:21:43.919657 10218 sys_catalog.cc:455] T 00000000000000000000000000000000 P 826f8c5f00154e17ac876679f3d43bca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "826f8c5f00154e17ac876679f3d43bca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "826f8c5f00154e17ac876679f3d43bca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46441 } } }
I20260929 03:21:43.919853 10218 sys_catalog.cc:458] T 00000000000000000000000000000000 P 826f8c5f00154e17ac876679f3d43bca [sys.catalog]: This master's current role is: LEADER
I20260929 03:21:43.919776 10219 sys_catalog.cc:458] T 00000000000000000000000000000000 P 826f8c5f00154e17ac876679f3d43bca [sys.catalog]: This master's current role is: LEADER
I20260929 03:21:43.920404 10227 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:21:43.920941 10227 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:21:43.922364 10227 catalog_manager.cc:1347] Generated new cluster ID: 8e01ece0b3774d41a5c7272813561361
I20260929 03:21:43.922415 10227 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:21:43.934844 10227 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:21:43.935037 10227 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:21:43.939131 10227 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 826f8c5f00154e17ac876679f3d43bca: Generated new TSK 0
I20260929 03:21:43.939335 10227 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 03:21:43.946012 21193 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuoaHbX/build/release/bin/kudu with pid 10140
2026-09-29T03:21:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1288 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790651852476208-21193-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:21:43 dist-test-slave-c9qh krb5kdc[10239](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:21:43 dist-test-slave-c9qh krb5kdc[10239](info): set up 2 sockets
Sep 29 03:21:43 dist-test-slave-c9qh krb5kdc[10239](info): commencing operation
krb5kdc: starting...
W20260929 03:21:44.968930 21193 mini_kdc.cc:121] Time spent starting KDC: real 1.005s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790651852476208-21193-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790651852476208-21193-0/krb5kdc/test-user.keytab.
Sep 29 03:21:45 dist-test-slave-c9qh krb5kdc[10239](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652105, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:21:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:21:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.178.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.178.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.178.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.126.keytab.
Entry for principal kudu/127.20.178.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.126.keytab.
I20260929 03:21:45.087235 21193 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuoaHbX/build/release/bin/kudu
/tmp/dist-test-taskuoaHbX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790651852476208-21193-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790651852476208-21193-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790651852476208-21193-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790651852476208-21193-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.178.126:39347
--webserver_interface=127.20.178.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.126.keytab
--principal=kudu/127.20.178.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.178.84:43249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.178.126:39347
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790651852476208-21193-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790651852476208-21193-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790651852476208-21193-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:21:45.176973 10252 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:21:45.177193 10252 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:21:45.177217 10252 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:21:45.178375 10252 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:21:45.178413 10252 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:21:45.178427 10252 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:21:45.178440 10252 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:21:45.178454 10252 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:21:45.179826 10252 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.178.84:43249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790651852476208-21193-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790651852476208-21193-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.178.126:39347
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.126.keytab
--principal=kudu/127.20.178.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.178.126:39347
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790651852476208-21193-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.178.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10252
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790651852476208-21193-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:21:45.180009 10252 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:21:45.180171 10252 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:21:45.185205 10258 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:21:45.185205 10257 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:21:45.185233 10252 server_base.cc:1034] running on GCE node
W20260929 03:21:45.185215 10260 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:21:45.185691 10252 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:21:45.185986 10252 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:21:45.187146 10252 hybrid_clock.cc:648] HybridClock initialized: now 1790652105187126 us; error 39 us; skew 500 ppm
Sep 29 03:21:45 dist-test-slave-c9qh krb5kdc[10239](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652105, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:21:45.189680 10252 init.cc:377] Logged in from keytab as kudu/127.20.178.126@KRBTEST.COM (short username kudu)
I20260929 03:21:45.190189 10252 webserver.cc:460] Webserver started at http://127.20.178.126:43399/ using document root <none> and password file <none>
I20260929 03:21:45.190426 10252 fs_manager.cc:359] Metadata directory not provided
I20260929 03:21:45.190495 10252 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:21:45.190631 10252 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:21:45.191650 10252 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790651852476208-21193-0/minicluster-data/master-0/data/instance:
uuid: "424b61ab8b9b4e1c8c1207dec5edbbce"
format_stamp: "Formatted at 2026-09-29 03:21:45 on dist-test-slave-c9qh"
server_key: "86c9d0d2e52e20c7454fe28cb53c1c55"
server_key_iv: "d715ce73cbc08b0fedea25ed3d3067f0"
server_key_version: "encryptionkey@0"
I20260929 03:21:45.192036 10252 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790651852476208-21193-0/minicluster-data/master-0/wal/instance:
uuid: "424b61ab8b9b4e1c8c1207dec5edbbce"
format_stamp: "Formatted at 2026-09-29 03:21:45 on dist-test-slave-c9qh"
server_key: "86c9d0d2e52e20c7454fe28cb53c1c55"
server_key_iv: "d715ce73cbc08b0fedea25ed3d3067f0"
server_key_version: "encryptionkey@0"
I20260929 03:21:45.193462 10252 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 03:21:45.194123 10267 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:21:45.194309 10252 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260929 03:21:45.194407 10252 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790651852476208-21193-0/minicluster-data/master-0/data,/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790651852476208-21193-0/minicluster-data/master-0/wal
uuid: "424b61ab8b9b4e1c8c1207dec5edbbce"
format_stamp: "Formatted at 2026-09-29 03:21:45 on dist-test-slave-c9qh"
server_key: "86c9d0d2e52e20c7454fe28cb53c1c55"
server_key_iv: "d715ce73cbc08b0fedea25ed3d3067f0"
server_key_version: "encryptionkey@0"
I20260929 03:21:45.194497 10252 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790651852476208-21193-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790651852476208-21193-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790651852476208-21193-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:21:45.209411 10252 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:21:45.211290 10252 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:21:45.211447 10252 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:21:45.215448 10252 rpc_server.cc:307] RPC server started. Bound to: 127.20.178.126:39347
I20260929 03:21:45.215479 10319 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.178.126:39347 every 8 connection(s)
I20260929 03:21:45.215814 10252 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790651852476208-21193-0/minicluster-data/master-0/data/info.pb
I20260929 03:21:45.216753 10320 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:21:45.218950 10320 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 424b61ab8b9b4e1c8c1207dec5edbbce: Bootstrap starting.
I20260929 03:21:45.219515 10320 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 424b61ab8b9b4e1c8c1207dec5edbbce: Neither blocks nor log segments found. Creating new log.
I20260929 03:21:45.219902 10320 log.cc:824] T 00000000000000000000000000000000 P 424b61ab8b9b4e1c8c1207dec5edbbce: Log is configured to *not* fsync() on all Append() calls
I20260929 03:21:45.220580 10320 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 424b61ab8b9b4e1c8c1207dec5edbbce: No bootstrap required, opened a new log
I20260929 03:21:45.222149 10320 raft_consensus.cc:348] T 00000000000000000000000000000000 P 424b61ab8b9b4e1c8c1207dec5edbbce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "424b61ab8b9b4e1c8c1207dec5edbbce" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 39347 } }
I20260929 03:21:45.222282 10320 raft_consensus.cc:374] T 00000000000000000000000000000000 P 424b61ab8b9b4e1c8c1207dec5edbbce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:21:45.222306 10320 raft_consensus.cc:729] T 00000000000000000000000000000000 P 424b61ab8b9b4e1c8c1207dec5edbbce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 424b61ab8b9b4e1c8c1207dec5edbbce, State: Initialized, Role: FOLLOWER
I20260929 03:21:45.222502 10320 consensus_queue.cc:260] T 00000000000000000000000000000000 P 424b61ab8b9b4e1c8c1207dec5edbbce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "424b61ab8b9b4e1c8c1207dec5edbbce" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 39347 } }
I20260929 03:21:45.222577 10320 raft_consensus.cc:388] T 00000000000000000000000000000000 P 424b61ab8b9b4e1c8c1207dec5edbbce [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:21:45.222582 21193 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuoaHbX/build/release/bin/kudu as pid 10252
I20260929 03:21:45.222599 10320 raft_consensus.cc:482] T 00000000000000000000000000000000 P 424b61ab8b9b4e1c8c1207dec5edbbce [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:21:45.222657 10320 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 424b61ab8b9b4e1c8c1207dec5edbbce [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:21:45.222684 21193 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790651852476208-21193-0/minicluster-data/master-0/wal/instance
I20260929 03:21:45.222855 21193 external_mini_cluster.cc:1428] Setting key ace3faf8cf040aed6f65c8a69f16367f
I20260929 03:21:45.223723 10320 raft_consensus.cc:504] T 00000000000000000000000000000000 P 424b61ab8b9b4e1c8c1207dec5edbbce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "424b61ab8b9b4e1c8c1207dec5edbbce" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 39347 } }
I20260929 03:21:45.223865 10320 leader_election.cc:302] T 00000000000000000000000000000000 P 424b61ab8b9b4e1c8c1207dec5edbbce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 424b61ab8b9b4e1c8c1207dec5edbbce; no voters:
I20260929 03:21:45.224035 10320 leader_election.cc:288] T 00000000000000000000000000000000 P 424b61ab8b9b4e1c8c1207dec5edbbce [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:21:45.224054 10325 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 424b61ab8b9b4e1c8c1207dec5edbbce [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:21:45.224228 10325 raft_consensus.cc:686] T 00000000000000000000000000000000 P 424b61ab8b9b4e1c8c1207dec5edbbce [term 1 LEADER]: Becoming Leader. State: Replica: 424b61ab8b9b4e1c8c1207dec5edbbce, State: Running, Role: LEADER
I20260929 03:21:45.224293 10320 sys_catalog.cc:564] T 00000000000000000000000000000000 P 424b61ab8b9b4e1c8c1207dec5edbbce [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:21:45.224345 10325 consensus_queue.cc:237] T 00000000000000000000000000000000 P 424b61ab8b9b4e1c8c1207dec5edbbce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "424b61ab8b9b4e1c8c1207dec5edbbce" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 39347 } }
I20260929 03:21:45.224799 10326 sys_catalog.cc:455] T 00000000000000000000000000000000 P 424b61ab8b9b4e1c8c1207dec5edbbce [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "424b61ab8b9b4e1c8c1207dec5edbbce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "424b61ab8b9b4e1c8c1207dec5edbbce" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 39347 } } }
I20260929 03:21:45.224887 10326 sys_catalog.cc:458] T 00000000000000000000000000000000 P 424b61ab8b9b4e1c8c1207dec5edbbce [sys.catalog]: This master's current role is: LEADER
I20260929 03:21:45.225071 10327 sys_catalog.cc:455] T 00000000000000000000000000000000 P 424b61ab8b9b4e1c8c1207dec5edbbce [sys.catalog]: SysCatalogTable state changed. Reason: New leader 424b61ab8b9b4e1c8c1207dec5edbbce. Latest consensus state: current_term: 1 leader_uuid: "424b61ab8b9b4e1c8c1207dec5edbbce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "424b61ab8b9b4e1c8c1207dec5edbbce" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 39347 } } }
I20260929 03:21:45.225169 10327 sys_catalog.cc:458] T 00000000000000000000000000000000 P 424b61ab8b9b4e1c8c1207dec5edbbce [sys.catalog]: This master's current role is: LEADER
I20260929 03:21:45.225324 10332 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:21:45.225847 10332 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:21:45.227417 10332 catalog_manager.cc:1347] Generated new cluster ID: 8226717eec144ed2a21d6b8ee64f6b5a
I20260929 03:21:45.227466 10332 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 29 03:21:45 dist-test-slave-c9qh krb5kdc[10239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652105, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.178.126@KRBTEST.COM
I20260929 03:21:45.234443 10324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:45.223345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53535 (local address 127.20.178.126:39347)
0929 03:21:45.223526 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0929 03:21:45.223533 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:21:45.223558 (+ 25us) server_negotiation.cc:407] Connection header received
0929 03:21:45.223645 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:21:45.223650 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:21:45.223690 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:21:45.223794 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:21:45.224675 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:45.226098 (+ 1423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:45.227160 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:45.227726 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:45.227823 (+ 97us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:21:45.230599 (+ 2776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:21:45.230603 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:21:45.230604 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:21:45.230616 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:21:45.232720 (+ 2104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:21:45.233129 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:21:45.233132 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:21:45.233133 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:21:45.233267 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:21:45.233506 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:21:45.233507 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:21:45.233507 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:21:45.233876 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:21:45.234039 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0929 03:21:45.234124 (+ 85us) server_negotiation.cc:299] Negotiation successful
0929 03:21:45.234225 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":100,"threads_started":1}
I20260929 03:21:45.235469 21193 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 03:21:45.246999 10332 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:21:45.247695 10332 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:21:45.248337 10324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:45.236865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34705 (local address 127.20.178.126:39347)
0929 03:21:45.236907 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0929 03:21:45.236909 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:21:45.240809 (+ 3900us) server_negotiation.cc:407] Connection header received
0929 03:21:45.240870 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:21:45.240871 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:21:45.240880 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:21:45.240940 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:21:45.241556 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:45.242275 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:45.243298 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:45.243605 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:45.243663 (+ 58us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:21:45.245164 (+ 1501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:21:45.245165 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:21:45.245165 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:21:45.245171 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:21:45.246743 (+ 1572us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:21:45.247185 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:21:45.247186 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:21:45.247186 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:21:45.247243 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:21:45.247488 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:21:45.247489 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:21:45.247489 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:21:45.247602 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:21:45.248143 (+ 541us) server_negotiation.cc:1035] Waiting for connection context
0929 03:21:45.248194 (+ 51us) server_negotiation.cc:299] Negotiation successful
0929 03:21:45.248256 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260929 03:21:45.254961 10332 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 424b61ab8b9b4e1c8c1207dec5edbbce: Generated new TSK 0
I20260929 03:21:45.255124 10332 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 03:21:45.261053 21193 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuoaHbX/build/release/bin/kudu with pid 10252
W20260929 03:21:45.265018 10237 connection.cc:535] client connection to 127.20.178.126:39347 recv error: Network error: failed to read from TLS socket (remote: 127.20.178.126:39347): Connection reset by peer (error 104)
2026-09-29T03:21:45Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1314 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790651852476208-21193-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:21:45 dist-test-slave-c9qh krb5kdc[10351](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:21:45 dist-test-slave-c9qh krb5kdc[10351](info): set up 2 sockets
Sep 29 03:21:45 dist-test-slave-c9qh krb5kdc[10351](info): commencing operation
krb5kdc: starting...
W20260929 03:21:46.288784 21193 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790651852476208-21193-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790651852476208-21193-0/krb5kdc/test-user.keytab.
Sep 29 03:21:46 dist-test-slave-c9qh krb5kdc[10351](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652106, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:21:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:21:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.178.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.178.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.178.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.126.keytab.
Entry for principal kudu/127.20.178.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.126.keytab.
I20260929 03:21:46.401687 21193 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuoaHbX/build/release/bin/kudu
/tmp/dist-test-taskuoaHbX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790651852476208-21193-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790651852476208-21193-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790651852476208-21193-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790651852476208-21193-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.178.126:36785
--webserver_interface=127.20.178.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.126.keytab
--principal=kudu/127.20.178.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.178.84:45567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.178.126:36785
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790651852476208-21193-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790651852476208-21193-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790651852476208-21193-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:21:46.480353 10364 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:21:46.480561 10364 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:21:46.480583 10364 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:21:46.481770 10364 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:21:46.481812 10364 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:21:46.481825 10364 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:21:46.481838 10364 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:21:46.481850 10364 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:21:46.483270 10364 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.178.84:45567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790651852476208-21193-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790651852476208-21193-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.178.126:36785
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.126.keytab
--principal=kudu/127.20.178.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.178.126:36785
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790651852476208-21193-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.178.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10364
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790651852476208-21193-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:21:46.483464 10364 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:21:46.483626 10364 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 03:21:46.488479 10364 server_base.cc:1034] running on GCE node
W20260929 03:21:46.488520 10372 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:21:46.488451 10370 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:21:46.488446 10369 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:21:46.488947 10364 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:21:46.489212 10364 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:21:46.490374 10364 hybrid_clock.cc:648] HybridClock initialized: now 1790652106490395 us; error 88 us; skew 500 ppm
Sep 29 03:21:46 dist-test-slave-c9qh krb5kdc[10351](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652106, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:21:46.492866 10364 init.cc:377] Logged in from keytab as kudu/127.20.178.126@KRBTEST.COM (short username kudu)
I20260929 03:21:46.493451 10364 webserver.cc:460] Webserver started at http://127.20.178.126:35647/ using document root <none> and password file <none>
I20260929 03:21:46.493680 10364 fs_manager.cc:359] Metadata directory not provided
I20260929 03:21:46.493746 10364 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:21:46.493878 10364 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:21:46.494906 10364 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790651852476208-21193-0/minicluster-data/master-0/data/instance:
uuid: "58fdcafe6b91424291fc581069f3189c"
format_stamp: "Formatted at 2026-09-29 03:21:46 on dist-test-slave-c9qh"
server_key: "19c021e39257db1b7fc7a70258347c4b"
server_key_iv: "527e0e271127ebaced47adbde4b86ca5"
server_key_version: "encryptionkey@0"
I20260929 03:21:46.495266 10364 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790651852476208-21193-0/minicluster-data/master-0/wal/instance:
uuid: "58fdcafe6b91424291fc581069f3189c"
format_stamp: "Formatted at 2026-09-29 03:21:46 on dist-test-slave-c9qh"
server_key: "19c021e39257db1b7fc7a70258347c4b"
server_key_iv: "527e0e271127ebaced47adbde4b86ca5"
server_key_version: "encryptionkey@0"
I20260929 03:21:46.496557 10364 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 03:21:46.497308 10379 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:21:46.497502 10364 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 03:21:46.497599 10364 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790651852476208-21193-0/minicluster-data/master-0/data,/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790651852476208-21193-0/minicluster-data/master-0/wal
uuid: "58fdcafe6b91424291fc581069f3189c"
format_stamp: "Formatted at 2026-09-29 03:21:46 on dist-test-slave-c9qh"
server_key: "19c021e39257db1b7fc7a70258347c4b"
server_key_iv: "527e0e271127ebaced47adbde4b86ca5"
server_key_version: "encryptionkey@0"
I20260929 03:21:46.497686 10364 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790651852476208-21193-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790651852476208-21193-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790651852476208-21193-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:21:46.534567 10364 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:21:46.536689 10364 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:21:46.536857 10364 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:21:46.541093 10364 rpc_server.cc:307] RPC server started. Bound to: 127.20.178.126:36785
I20260929 03:21:46.541131 10431 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.178.126:36785 every 8 connection(s)
I20260929 03:21:46.541520 10364 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790651852476208-21193-0/minicluster-data/master-0/data/info.pb
I20260929 03:21:46.542362 10432 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:21:46.544625 10432 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58fdcafe6b91424291fc581069f3189c: Bootstrap starting.
I20260929 03:21:46.545291 10432 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 58fdcafe6b91424291fc581069f3189c: Neither blocks nor log segments found. Creating new log.
I20260929 03:21:46.545583 10432 log.cc:824] T 00000000000000000000000000000000 P 58fdcafe6b91424291fc581069f3189c: Log is configured to *not* fsync() on all Append() calls
I20260929 03:21:46.546289 10432 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58fdcafe6b91424291fc581069f3189c: No bootstrap required, opened a new log
I20260929 03:21:46.546311 21193 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuoaHbX/build/release/bin/kudu as pid 10364
I20260929 03:21:46.546455 21193 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790651852476208-21193-0/minicluster-data/master-0/wal/instance
I20260929 03:21:46.546641 21193 external_mini_cluster.cc:1428] Setting key 33ea0bc9b87df13155ed8d28721e5661
I20260929 03:21:46.548185 10432 raft_consensus.cc:348] T 00000000000000000000000000000000 P 58fdcafe6b91424291fc581069f3189c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58fdcafe6b91424291fc581069f3189c" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 36785 } }
I20260929 03:21:46.548354 10432 raft_consensus.cc:374] T 00000000000000000000000000000000 P 58fdcafe6b91424291fc581069f3189c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:21:46.548388 10432 raft_consensus.cc:729] T 00000000000000000000000000000000 P 58fdcafe6b91424291fc581069f3189c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58fdcafe6b91424291fc581069f3189c, State: Initialized, Role: FOLLOWER
I20260929 03:21:46.548512 10432 consensus_queue.cc:260] T 00000000000000000000000000000000 P 58fdcafe6b91424291fc581069f3189c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58fdcafe6b91424291fc581069f3189c" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 36785 } }
I20260929 03:21:46.548594 10432 raft_consensus.cc:388] T 00000000000000000000000000000000 P 58fdcafe6b91424291fc581069f3189c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:21:46.548661 10432 raft_consensus.cc:482] T 00000000000000000000000000000000 P 58fdcafe6b91424291fc581069f3189c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:21:46.548714 10432 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 58fdcafe6b91424291fc581069f3189c [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:21:46.549243 10432 raft_consensus.cc:504] T 00000000000000000000000000000000 P 58fdcafe6b91424291fc581069f3189c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58fdcafe6b91424291fc581069f3189c" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 36785 } }
I20260929 03:21:46.549329 10432 leader_election.cc:302] T 00000000000000000000000000000000 P 58fdcafe6b91424291fc581069f3189c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 58fdcafe6b91424291fc581069f3189c; no voters:
I20260929 03:21:46.549523 10432 leader_election.cc:288] T 00000000000000000000000000000000 P 58fdcafe6b91424291fc581069f3189c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:21:46.549552 10437 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 58fdcafe6b91424291fc581069f3189c [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:21:46.549655 10437 raft_consensus.cc:686] T 00000000000000000000000000000000 P 58fdcafe6b91424291fc581069f3189c [term 1 LEADER]: Becoming Leader. State: Replica: 58fdcafe6b91424291fc581069f3189c, State: Running, Role: LEADER
I20260929 03:21:46.549785 10432 sys_catalog.cc:564] T 00000000000000000000000000000000 P 58fdcafe6b91424291fc581069f3189c [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:21:46.549795 10437 consensus_queue.cc:237] T 00000000000000000000000000000000 P 58fdcafe6b91424291fc581069f3189c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58fdcafe6b91424291fc581069f3189c" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 36785 } }
I20260929 03:21:46.550369 10439 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58fdcafe6b91424291fc581069f3189c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 58fdcafe6b91424291fc581069f3189c. Latest consensus state: current_term: 1 leader_uuid: "58fdcafe6b91424291fc581069f3189c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58fdcafe6b91424291fc581069f3189c" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 36785 } } }
I20260929 03:21:46.550455 10439 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58fdcafe6b91424291fc581069f3189c [sys.catalog]: This master's current role is: LEADER
I20260929 03:21:46.550657 10438 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58fdcafe6b91424291fc581069f3189c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "58fdcafe6b91424291fc581069f3189c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58fdcafe6b91424291fc581069f3189c" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 36785 } } }
I20260929 03:21:46.550755 10438 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58fdcafe6b91424291fc581069f3189c [sys.catalog]: This master's current role is: LEADER
I20260929 03:21:46.550961 10446 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:21:46.551460 10446 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:21:46.553318 10446 catalog_manager.cc:1347] Generated new cluster ID: b08194d6fcef41bca76b64f439bafeaf
I20260929 03:21:46.553419 10446 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:21:46.553602 10436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:46.547177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37737 (local address 127.20.178.126:36785)
0929 03:21:46.547349 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0929 03:21:46.547359 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 03:21:46.547392 (+ 33us) server_negotiation.cc:407] Connection header received
0929 03:21:46.548170 (+ 778us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:21:46.548176 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:21:46.548223 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:21:46.548550 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:21:46.548784 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:46.550313 (+ 1529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:46.551406 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:46.551983 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:46.552092 (+ 109us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:21:46.552853 (+ 761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:21:46.552856 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:21:46.552857 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:21:46.553071 (+ 214us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:21:46.553116 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:21:46.553127 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:21:46.553223 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0929 03:21:46.553355 (+ 132us) server_negotiation.cc:299] Negotiation successful
0929 03:21:46.553381 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":80,"threads_started":1}
I20260929 03:21:46.554534 21193 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 03:21:46.559501 10436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:46.555938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44729 (local address 127.20.178.126:36785)
0929 03:21:46.555973 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0929 03:21:46.555975 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:21:46.556105 (+ 130us) server_negotiation.cc:407] Connection header received
0929 03:21:46.556162 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:21:46.556163 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:21:46.556172 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:21:46.556225 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:21:46.556338 (+ 113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:46.557078 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:46.558151 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:46.558482 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:46.558538 (+ 56us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:21:46.559238 (+ 700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:21:46.559239 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:21:46.559239 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:21:46.559243 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:21:46.559256 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:21:46.559260 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:21:46.559307 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0929 03:21:46.559419 (+ 112us) server_negotiation.cc:299] Negotiation successful
0929 03:21:46.559431 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260929 03:21:46.560830 10446 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:21:46.561481 10446 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:21:46.572331 10446 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 58fdcafe6b91424291fc581069f3189c: Generated new TSK 0
I20260929 03:21:46.572515 10446 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 03:21:46.580591 21193 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuoaHbX/build/release/bin/kudu with pid 10364
W20260929 03:21:46.584334 10349 connection.cc:535] client connection to 127.20.178.126:36785 recv error: Network error: failed to read from TLS socket (remote: 127.20.178.126:36785): Connection reset by peer (error 104)
2026-09-29T03:21:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1319 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790651852476208-21193-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:21:46 dist-test-slave-c9qh krb5kdc[10463](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:21:46 dist-test-slave-c9qh krb5kdc[10463](info): set up 2 sockets
Sep 29 03:21:46 dist-test-slave-c9qh krb5kdc[10463](info): commencing operation
krb5kdc: starting...
W20260929 03:21:47.628684 21193 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790651852476208-21193-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790651852476208-21193-0/krb5kdc/test-user.keytab.
Sep 29 03:21:47 dist-test-slave-c9qh krb5kdc[10463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652107, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:21:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:21:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.178.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.178.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.178.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.126.keytab.
Entry for principal kudu/127.20.178.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.126.keytab.
I20260929 03:21:47.741369 21193 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuoaHbX/build/release/bin/kudu
/tmp/dist-test-taskuoaHbX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790651852476208-21193-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790651852476208-21193-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790651852476208-21193-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790651852476208-21193-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.178.126:45927
--webserver_interface=127.20.178.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.126.keytab
--principal=kudu/127.20.178.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.178.84:41655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.178.126:45927
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790651852476208-21193-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790651852476208-21193-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790651852476208-21193-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:21:47.839071 10476 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:21:47.839393 10476 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:21:47.839452 10476 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:21:47.841763 10476 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:21:47.841826 10476 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:21:47.841856 10476 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:21:47.841938 10476 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:21:47.841975 10476 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:21:47.844805 10476 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.178.84:41655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790651852476208-21193-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790651852476208-21193-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.178.126:45927
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.126.keytab
--principal=kudu/127.20.178.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.178.126:45927
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790651852476208-21193-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.178.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10476
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790651852476208-21193-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:21:47.845091 10476 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:21:47.845386 10476 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:21:47.850832 10481 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:21:47.850960 10476 server_base.cc:1034] running on GCE node
W20260929 03:21:47.850845 10482 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:21:47.851015 10484 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:21:47.851279 10476 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:21:47.851601 10476 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:21:47.852687 10476 hybrid_clock.cc:648] HybridClock initialized: now 1790652107852670 us; error 34 us; skew 500 ppm
Sep 29 03:21:47 dist-test-slave-c9qh krb5kdc[10463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652107, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:21:47.855154 10476 init.cc:377] Logged in from keytab as kudu/127.20.178.126@KRBTEST.COM (short username kudu)
I20260929 03:21:47.855615 10476 webserver.cc:460] Webserver started at http://127.20.178.126:43433/ using document root <none> and password file <none>
I20260929 03:21:47.855845 10476 fs_manager.cc:359] Metadata directory not provided
I20260929 03:21:47.855908 10476 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:21:47.856058 10476 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:21:47.857221 10476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790651852476208-21193-0/minicluster-data/master-0/data/instance:
uuid: "41f573e66f034c61a627935e0d997c66"
format_stamp: "Formatted at 2026-09-29 03:21:47 on dist-test-slave-c9qh"
server_key: "d89c51f6f1054a71cce1e4142903cde7"
server_key_iv: "de0c9374256d124186f2608de3048ea3"
server_key_version: "encryptionkey@0"
I20260929 03:21:47.857584 10476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790651852476208-21193-0/minicluster-data/master-0/wal/instance:
uuid: "41f573e66f034c61a627935e0d997c66"
format_stamp: "Formatted at 2026-09-29 03:21:47 on dist-test-slave-c9qh"
server_key: "d89c51f6f1054a71cce1e4142903cde7"
server_key_iv: "de0c9374256d124186f2608de3048ea3"
server_key_version: "encryptionkey@0"
I20260929 03:21:47.858916 10476 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 03:21:47.859642 10491 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:21:47.859807 10476 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 03:21:47.859906 10476 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790651852476208-21193-0/minicluster-data/master-0/data,/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790651852476208-21193-0/minicluster-data/master-0/wal
uuid: "41f573e66f034c61a627935e0d997c66"
format_stamp: "Formatted at 2026-09-29 03:21:47 on dist-test-slave-c9qh"
server_key: "d89c51f6f1054a71cce1e4142903cde7"
server_key_iv: "de0c9374256d124186f2608de3048ea3"
server_key_version: "encryptionkey@0"
I20260929 03:21:47.859977 10476 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790651852476208-21193-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790651852476208-21193-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790651852476208-21193-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:21:47.863270 10476 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:21:47.865536 10476 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:21:47.865685 10476 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:21:47.869526 10476 rpc_server.cc:307] RPC server started. Bound to: 127.20.178.126:45927
I20260929 03:21:47.869554 10543 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.178.126:45927 every 8 connection(s)
I20260929 03:21:47.869866 10476 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790651852476208-21193-0/minicluster-data/master-0/data/info.pb
I20260929 03:21:47.870787 10544 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:21:47.873015 10544 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41f573e66f034c61a627935e0d997c66: Bootstrap starting.
I20260929 03:21:47.873529 10544 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 41f573e66f034c61a627935e0d997c66: Neither blocks nor log segments found. Creating new log.
I20260929 03:21:47.873755 10544 log.cc:824] T 00000000000000000000000000000000 P 41f573e66f034c61a627935e0d997c66: Log is configured to *not* fsync() on all Append() calls
I20260929 03:21:47.874363 10544 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41f573e66f034c61a627935e0d997c66: No bootstrap required, opened a new log
I20260929 03:21:47.875957 21193 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuoaHbX/build/release/bin/kudu as pid 10476
I20260929 03:21:47.876071 21193 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790651852476208-21193-0/minicluster-data/master-0/wal/instance
I20260929 03:21:47.875974 10544 raft_consensus.cc:348] T 00000000000000000000000000000000 P 41f573e66f034c61a627935e0d997c66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41f573e66f034c61a627935e0d997c66" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 45927 } }
I20260929 03:21:47.876164 10544 raft_consensus.cc:374] T 00000000000000000000000000000000 P 41f573e66f034c61a627935e0d997c66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:21:47.876209 10544 raft_consensus.cc:729] T 00000000000000000000000000000000 P 41f573e66f034c61a627935e0d997c66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41f573e66f034c61a627935e0d997c66, State: Initialized, Role: FOLLOWER
I20260929 03:21:47.876276 21193 external_mini_cluster.cc:1428] Setting key f2b67bdcdb2f605be6cbce3e0329e7cd
I20260929 03:21:47.876364 10544 consensus_queue.cc:260] T 00000000000000000000000000000000 P 41f573e66f034c61a627935e0d997c66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41f573e66f034c61a627935e0d997c66" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 45927 } }
I20260929 03:21:47.876453 10544 raft_consensus.cc:388] T 00000000000000000000000000000000 P 41f573e66f034c61a627935e0d997c66 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:21:47.876497 10544 raft_consensus.cc:482] T 00000000000000000000000000000000 P 41f573e66f034c61a627935e0d997c66 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:21:47.876546 10544 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 41f573e66f034c61a627935e0d997c66 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:21:47.877501 10544 raft_consensus.cc:504] T 00000000000000000000000000000000 P 41f573e66f034c61a627935e0d997c66 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41f573e66f034c61a627935e0d997c66" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 45927 } }
I20260929 03:21:47.877609 10544 leader_election.cc:302] T 00000000000000000000000000000000 P 41f573e66f034c61a627935e0d997c66 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 41f573e66f034c61a627935e0d997c66; no voters:
I20260929 03:21:47.877763 10544 leader_election.cc:288] T 00000000000000000000000000000000 P 41f573e66f034c61a627935e0d997c66 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:21:47.877810 10549 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 41f573e66f034c61a627935e0d997c66 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:21:47.877983 10549 raft_consensus.cc:686] T 00000000000000000000000000000000 P 41f573e66f034c61a627935e0d997c66 [term 1 LEADER]: Becoming Leader. State: Replica: 41f573e66f034c61a627935e0d997c66, State: Running, Role: LEADER
I20260929 03:21:47.878072 10544 sys_catalog.cc:564] T 00000000000000000000000000000000 P 41f573e66f034c61a627935e0d997c66 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:21:47.878126 10549 consensus_queue.cc:237] T 00000000000000000000000000000000 P 41f573e66f034c61a627935e0d997c66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41f573e66f034c61a627935e0d997c66" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 45927 } }
I20260929 03:21:47.878577 10551 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41f573e66f034c61a627935e0d997c66 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 41f573e66f034c61a627935e0d997c66. Latest consensus state: current_term: 1 leader_uuid: "41f573e66f034c61a627935e0d997c66" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41f573e66f034c61a627935e0d997c66" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 45927 } } }
I20260929 03:21:47.878623 10548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:47.876733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44675 (local address 127.20.178.126:45927)
0929 03:21:47.876916 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0929 03:21:47.876926 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 03:21:47.876958 (+ 32us) server_negotiation.cc:407] Connection header received
0929 03:21:47.877096 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:21:47.877101 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:21:47.877124 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:21:47.877251 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:21:47.878004 (+ 753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:21:47.878008 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:21:47.878008 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:21:47.878013 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:21:47.878058 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:21:47.878074 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:21:47.878139 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0929 03:21:47.878354 (+ 215us) server_negotiation.cc:299] Negotiation successful
0929 03:21:47.878398 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":81,"threads_started":1}
I20260929 03:21:47.878660 10551 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41f573e66f034c61a627935e0d997c66 [sys.catalog]: This master's current role is: LEADER
I20260929 03:21:47.878811 10550 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41f573e66f034c61a627935e0d997c66 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "41f573e66f034c61a627935e0d997c66" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41f573e66f034c61a627935e0d997c66" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 45927 } } }
I20260929 03:21:47.878872 10550 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41f573e66f034c61a627935e0d997c66 [sys.catalog]: This master's current role is: LEADER
I20260929 03:21:47.878906 10558 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:21:47.879419 10558 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:21:47.879940 21193 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 03:21:47.881436 10558 catalog_manager.cc:1347] Generated new cluster ID: f9311a147b0346cf942c989f8a126a15
I20260929 03:21:47.881488 10558 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:21:47.885713 10548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:47.884724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59181 (local address 127.20.178.126:45927)
0929 03:21:47.884762 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0929 03:21:47.884764 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:21:47.884770 (+ 6us) server_negotiation.cc:407] Connection header received
0929 03:21:47.884791 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:21:47.884792 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:21:47.884796 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:21:47.884851 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:21:47.885509 (+ 658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:21:47.885510 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:21:47.885510 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:21:47.885514 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:21:47.885528 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:21:47.885532 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:21:47.885577 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0929 03:21:47.885638 (+ 61us) server_negotiation.cc:299] Negotiation successful
0929 03:21:47.885650 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260929 03:21:47.894573 10558 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:21:47.894757 10558 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:21:47.905941 10558 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 41f573e66f034c61a627935e0d997c66: Generated new TSK 0
I20260929 03:21:47.906136 10558 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 03:21:47.916193 21193 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuoaHbX/build/release/bin/kudu with pid 10476
2026-09-29T03:21:47Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1336 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790651852476208-21193-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:21:47 dist-test-slave-c9qh krb5kdc[10575](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:21:47 dist-test-slave-c9qh krb5kdc[10575](info): set up 2 sockets
Sep 29 03:21:47 dist-test-slave-c9qh krb5kdc[10575](info): commencing operation
krb5kdc: starting...
W20260929 03:21:48.972904 21193 mini_kdc.cc:121] Time spent starting KDC: real 1.038s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790651852476208-21193-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790651852476208-21193-0/krb5kdc/test-user.keytab.
Sep 29 03:21:49 dist-test-slave-c9qh krb5kdc[10575](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652109, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:21:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:21:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.178.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.178.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.178.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.126.keytab.
Entry for principal kudu/127.20.178.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.126.keytab.
I20260929 03:21:49.089429 21193 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuoaHbX/build/release/bin/kudu
/tmp/dist-test-taskuoaHbX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790651852476208-21193-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790651852476208-21193-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790651852476208-21193-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790651852476208-21193-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.178.126:36341
--webserver_interface=127.20.178.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.126.keytab
--principal=kudu/127.20.178.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.178.84:39445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.178.126:36341
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790651852476208-21193-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790651852476208-21193-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790651852476208-21193-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:21:49.168395 10588 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:21:49.168596 10588 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:21:49.168619 10588 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:21:49.169818 10588 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:21:49.169862 10588 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:21:49.169875 10588 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:21:49.169888 10588 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:21:49.169900 10588 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:21:49.171367 10588 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.178.84:39445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790651852476208-21193-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790651852476208-21193-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.178.126:36341
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.126.keytab
--principal=kudu/127.20.178.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.178.126:36341
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790651852476208-21193-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.178.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10588
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790651852476208-21193-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:21:49.171613 10588 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:21:49.171826 10588 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:21:49.176508 10596 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:21:49.176462 10594 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:21:49.176473 10593 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:21:49.176748 10588 server_base.cc:1034] running on GCE node
I20260929 03:21:49.176994 10588 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:21:49.177268 10588 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:21:49.178402 10588 hybrid_clock.cc:648] HybridClock initialized: now 1790652109178385 us; error 36 us; skew 500 ppm
Sep 29 03:21:49 dist-test-slave-c9qh krb5kdc[10575](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652109, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:21:49.185873 10588 init.cc:377] Logged in from keytab as kudu/127.20.178.126@KRBTEST.COM (short username kudu)
I20260929 03:21:49.186331 10588 webserver.cc:460] Webserver started at http://127.20.178.126:42483/ using document root <none> and password file <none>
I20260929 03:21:49.186556 10588 fs_manager.cc:359] Metadata directory not provided
I20260929 03:21:49.186619 10588 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:21:49.186760 10588 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:21:49.187810 10588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790651852476208-21193-0/minicluster-data/master-0/data/instance:
uuid: "1392fcea9d2449e49cb4e3ccb3f239a7"
format_stamp: "Formatted at 2026-09-29 03:21:49 on dist-test-slave-c9qh"
server_key: "98617952e55147e595f1386d4148ac27"
server_key_iv: "d1ab81d6ef78686c99eb6cb75f367689"
server_key_version: "encryptionkey@0"
I20260929 03:21:49.188154 10588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790651852476208-21193-0/minicluster-data/master-0/wal/instance:
uuid: "1392fcea9d2449e49cb4e3ccb3f239a7"
format_stamp: "Formatted at 2026-09-29 03:21:49 on dist-test-slave-c9qh"
server_key: "98617952e55147e595f1386d4148ac27"
server_key_iv: "d1ab81d6ef78686c99eb6cb75f367689"
server_key_version: "encryptionkey@0"
I20260929 03:21:49.189524 10588 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 03:21:49.190202 10603 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:21:49.190364 10588 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260929 03:21:49.190469 10588 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790651852476208-21193-0/minicluster-data/master-0/data,/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790651852476208-21193-0/minicluster-data/master-0/wal
uuid: "1392fcea9d2449e49cb4e3ccb3f239a7"
format_stamp: "Formatted at 2026-09-29 03:21:49 on dist-test-slave-c9qh"
server_key: "98617952e55147e595f1386d4148ac27"
server_key_iv: "d1ab81d6ef78686c99eb6cb75f367689"
server_key_version: "encryptionkey@0"
I20260929 03:21:49.190538 10588 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790651852476208-21193-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790651852476208-21193-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790651852476208-21193-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:21:49.212457 10588 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:21:49.214435 10588 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:21:49.214596 10588 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:21:49.218494 10588 rpc_server.cc:307] RPC server started. Bound to: 127.20.178.126:36341
I20260929 03:21:49.218544 10655 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.178.126:36341 every 8 connection(s)
I20260929 03:21:49.218894 10588 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790651852476208-21193-0/minicluster-data/master-0/data/info.pb
I20260929 03:21:49.219712 10656 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:21:49.221948 10656 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1392fcea9d2449e49cb4e3ccb3f239a7: Bootstrap starting.
I20260929 03:21:49.222478 10656 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1392fcea9d2449e49cb4e3ccb3f239a7: Neither blocks nor log segments found. Creating new log.
I20260929 03:21:49.222708 10656 log.cc:824] T 00000000000000000000000000000000 P 1392fcea9d2449e49cb4e3ccb3f239a7: Log is configured to *not* fsync() on all Append() calls
I20260929 03:21:49.223337 10656 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1392fcea9d2449e49cb4e3ccb3f239a7: No bootstrap required, opened a new log
I20260929 03:21:49.223800 21193 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuoaHbX/build/release/bin/kudu as pid 10588
I20260929 03:21:49.223910 21193 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790651852476208-21193-0/minicluster-data/master-0/wal/instance
I20260929 03:21:49.224090 21193 external_mini_cluster.cc:1428] Setting key b24b5378cf7b6dcfbfdb12476b62860d
I20260929 03:21:49.225785 10656 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1392fcea9d2449e49cb4e3ccb3f239a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1392fcea9d2449e49cb4e3ccb3f239a7" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 36341 } }
I20260929 03:21:49.225976 10656 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1392fcea9d2449e49cb4e3ccb3f239a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:21:49.226022 10656 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1392fcea9d2449e49cb4e3ccb3f239a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1392fcea9d2449e49cb4e3ccb3f239a7, State: Initialized, Role: FOLLOWER
I20260929 03:21:49.226156 10656 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1392fcea9d2449e49cb4e3ccb3f239a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1392fcea9d2449e49cb4e3ccb3f239a7" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 36341 } }
I20260929 03:21:49.226243 10656 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1392fcea9d2449e49cb4e3ccb3f239a7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:21:49.226284 10656 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1392fcea9d2449e49cb4e3ccb3f239a7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:21:49.226337 10656 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1392fcea9d2449e49cb4e3ccb3f239a7 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:21:49.226969 10656 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1392fcea9d2449e49cb4e3ccb3f239a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1392fcea9d2449e49cb4e3ccb3f239a7" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 36341 } }
I20260929 03:21:49.227056 10656 leader_election.cc:302] T 00000000000000000000000000000000 P 1392fcea9d2449e49cb4e3ccb3f239a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1392fcea9d2449e49cb4e3ccb3f239a7; no voters:
I20260929 03:21:49.227237 10656 leader_election.cc:288] T 00000000000000000000000000000000 P 1392fcea9d2449e49cb4e3ccb3f239a7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:21:49.227269 10661 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1392fcea9d2449e49cb4e3ccb3f239a7 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:21:49.227378 10661 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1392fcea9d2449e49cb4e3ccb3f239a7 [term 1 LEADER]: Becoming Leader. State: Replica: 1392fcea9d2449e49cb4e3ccb3f239a7, State: Running, Role: LEADER
I20260929 03:21:49.227494 10656 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1392fcea9d2449e49cb4e3ccb3f239a7 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:21:49.227521 10661 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1392fcea9d2449e49cb4e3ccb3f239a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1392fcea9d2449e49cb4e3ccb3f239a7" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 36341 } }
I20260929 03:21:49.227882 10661 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1392fcea9d2449e49cb4e3ccb3f239a7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1392fcea9d2449e49cb4e3ccb3f239a7. Latest consensus state: current_term: 1 leader_uuid: "1392fcea9d2449e49cb4e3ccb3f239a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1392fcea9d2449e49cb4e3ccb3f239a7" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 36341 } } }
I20260929 03:21:49.227892 10662 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1392fcea9d2449e49cb4e3ccb3f239a7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1392fcea9d2449e49cb4e3ccb3f239a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1392fcea9d2449e49cb4e3ccb3f239a7" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 36341 } } }
I20260929 03:21:49.228106 10662 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1392fcea9d2449e49cb4e3ccb3f239a7 [sys.catalog]: This master's current role is: LEADER
I20260929 03:21:49.228282 10661 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1392fcea9d2449e49cb4e3ccb3f239a7 [sys.catalog]: This master's current role is: LEADER
I20260929 03:21:49.228327 10670 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:21:49.228900 10670 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:21:49.230424 10670 catalog_manager.cc:1347] Generated new cluster ID: e923852587504240adf9296d2dd3b0a8
I20260929 03:21:49.230470 10670 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 29 03:21:49 dist-test-slave-c9qh krb5kdc[10575](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652109, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.178.126@KRBTEST.COM
I20260929 03:21:49.236197 10660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:49.224614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56461 (local address 127.20.178.126:36341)
0929 03:21:49.224794 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0929 03:21:49.224801 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:21:49.224828 (+ 27us) server_negotiation.cc:407] Connection header received
0929 03:21:49.225728 (+ 900us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:21:49.225735 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:21:49.225788 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:21:49.225903 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:21:49.226692 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:49.227878 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:49.229001 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:49.229668 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:49.229789 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:21:49.232487 (+ 2698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:21:49.232491 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:21:49.232492 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:21:49.232503 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:21:49.234459 (+ 1956us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:21:49.234938 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:21:49.234942 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:21:49.234942 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:21:49.235042 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:21:49.235251 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:21:49.235252 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:21:49.235252 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:21:49.235535 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:21:49.235674 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0929 03:21:49.235791 (+ 117us) server_negotiation.cc:299] Negotiation successful
0929 03:21:49.236003 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":79,"threads_started":1}
I20260929 03:21:49.237095 21193 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 03:21:49.245494 10660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:49.238465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56073 (local address 127.20.178.126:36341)
0929 03:21:49.238501 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0929 03:21:49.238503 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:21:49.238666 (+ 163us) server_negotiation.cc:407] Connection header received
0929 03:21:49.238698 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:21:49.238699 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:21:49.238709 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:21:49.238776 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:21:49.239327 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:49.240083 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:49.241081 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:49.241419 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:49.241500 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:21:49.242788 (+ 1288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:21:49.242790 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:21:49.242790 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:21:49.242797 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:21:49.244378 (+ 1581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:21:49.244744 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:21:49.244745 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:21:49.244745 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:21:49.244803 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:21:49.244990 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:21:49.244991 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:21:49.244992 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:21:49.245120 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:21:49.245216 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0929 03:21:49.245323 (+ 107us) server_negotiation.cc:299] Negotiation successful
0929 03:21:49.245395 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260929 03:21:49.246892 10670 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:21:49.247335 10670 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:21:49.250777 10670 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1392fcea9d2449e49cb4e3ccb3f239a7: Generated new TSK 0
I20260929 03:21:49.250936 10670 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 03:21:49.256778 21193 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuoaHbX/build/release/bin/kudu with pid 10588
2026-09-29T03:21:49Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1339 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790651852476208-21193-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:21:49 dist-test-slave-c9qh krb5kdc[10687](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:21:49 dist-test-slave-c9qh krb5kdc[10687](info): set up 2 sockets
Sep 29 03:21:49 dist-test-slave-c9qh krb5kdc[10687](info): commencing operation
krb5kdc: starting...
W20260929 03:21:50.292881 21193 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790651852476208-21193-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790651852476208-21193-0/krb5kdc/test-user.keytab.
Sep 29 03:21:50 dist-test-slave-c9qh krb5kdc[10687](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652110, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:21:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:21:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.178.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.178.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.178.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.126.keytab.
Entry for principal kudu/127.20.178.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.126.keytab.
I20260929 03:21:50.405975 21193 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuoaHbX/build/release/bin/kudu
/tmp/dist-test-taskuoaHbX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790651852476208-21193-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790651852476208-21193-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790651852476208-21193-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790651852476208-21193-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.178.126:38457
--webserver_interface=127.20.178.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.126.keytab
--principal=kudu/127.20.178.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.178.84:39053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.178.126:38457
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790651852476208-21193-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790651852476208-21193-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790651852476208-21193-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:21:50.493394 10700 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:21:50.493628 10700 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:21:50.493662 10700 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:21:50.494801 10700 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:21:50.494866 10700 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:21:50.494889 10700 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:21:50.494918 10700 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:21:50.494946 10700 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:21:50.496398 10700 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.178.84:39053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790651852476208-21193-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790651852476208-21193-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.178.126:38457
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.126.keytab
--principal=kudu/127.20.178.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.178.126:38457
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790651852476208-21193-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.178.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10700
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790651852476208-21193-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:21:50.496692 10700 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:21:50.496914 10700 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:21:50.501741 10706 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:21:50.501782 10700 server_base.cc:1034] running on GCE node
W20260929 03:21:50.501736 10708 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:21:50.501716 10705 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:21:50.502259 10700 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:21:50.502545 10700 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:21:50.503701 10700 hybrid_clock.cc:648] HybridClock initialized: now 1790652110503700 us; error 35 us; skew 500 ppm
Sep 29 03:21:50 dist-test-slave-c9qh krb5kdc[10687](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652110, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:21:50.506139 10700 init.cc:377] Logged in from keytab as kudu/127.20.178.126@KRBTEST.COM (short username kudu)
I20260929 03:21:50.506608 10700 webserver.cc:460] Webserver started at http://127.20.178.126:40615/ using document root <none> and password file <none>
I20260929 03:21:50.506835 10700 fs_manager.cc:359] Metadata directory not provided
I20260929 03:21:50.506898 10700 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:21:50.507040 10700 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:21:50.508100 10700 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790651852476208-21193-0/minicluster-data/master-0/data/instance:
uuid: "a7373ecfc6ca4b93b4f91158b42ad354"
format_stamp: "Formatted at 2026-09-29 03:21:50 on dist-test-slave-c9qh"
server_key: "4e8d201ef456d8f4a70e809d5ee3c853"
server_key_iv: "cf933d5273aebfbd09ee110c978e1e79"
server_key_version: "encryptionkey@0"
I20260929 03:21:50.508462 10700 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790651852476208-21193-0/minicluster-data/master-0/wal/instance:
uuid: "a7373ecfc6ca4b93b4f91158b42ad354"
format_stamp: "Formatted at 2026-09-29 03:21:50 on dist-test-slave-c9qh"
server_key: "4e8d201ef456d8f4a70e809d5ee3c853"
server_key_iv: "cf933d5273aebfbd09ee110c978e1e79"
server_key_version: "encryptionkey@0"
I20260929 03:21:50.509835 10700 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260929 03:21:50.510589 10715 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:21:50.510759 10700 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260929 03:21:50.510854 10700 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790651852476208-21193-0/minicluster-data/master-0/data,/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790651852476208-21193-0/minicluster-data/master-0/wal
uuid: "a7373ecfc6ca4b93b4f91158b42ad354"
format_stamp: "Formatted at 2026-09-29 03:21:50 on dist-test-slave-c9qh"
server_key: "4e8d201ef456d8f4a70e809d5ee3c853"
server_key_iv: "cf933d5273aebfbd09ee110c978e1e79"
server_key_version: "encryptionkey@0"
I20260929 03:21:50.510936 10700 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790651852476208-21193-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790651852476208-21193-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790651852476208-21193-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:21:50.529536 10700 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:21:50.531525 10700 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:21:50.531687 10700 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:21:50.535849 10700 rpc_server.cc:307] RPC server started. Bound to: 127.20.178.126:38457
I20260929 03:21:50.535902 10767 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.178.126:38457 every 8 connection(s)
I20260929 03:21:50.536263 10700 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790651852476208-21193-0/minicluster-data/master-0/data/info.pb
I20260929 03:21:50.537194 10768 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:21:50.539348 10768 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a7373ecfc6ca4b93b4f91158b42ad354: Bootstrap starting.
I20260929 03:21:50.539881 10768 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a7373ecfc6ca4b93b4f91158b42ad354: Neither blocks nor log segments found. Creating new log.
I20260929 03:21:50.540122 10768 log.cc:824] T 00000000000000000000000000000000 P a7373ecfc6ca4b93b4f91158b42ad354: Log is configured to *not* fsync() on all Append() calls
I20260929 03:21:50.540838 10768 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a7373ecfc6ca4b93b4f91158b42ad354: No bootstrap required, opened a new log
I20260929 03:21:50.540854 21193 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuoaHbX/build/release/bin/kudu as pid 10700
I20260929 03:21:50.540975 21193 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790651852476208-21193-0/minicluster-data/master-0/wal/instance
I20260929 03:21:50.541139 21193 external_mini_cluster.cc:1428] Setting key 64a70a34de7cf2de8d24aab774c9e279
I20260929 03:21:50.542714 10768 raft_consensus.cc:348] T 00000000000000000000000000000000 P a7373ecfc6ca4b93b4f91158b42ad354 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7373ecfc6ca4b93b4f91158b42ad354" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 38457 } }
I20260929 03:21:50.542886 10768 raft_consensus.cc:374] T 00000000000000000000000000000000 P a7373ecfc6ca4b93b4f91158b42ad354 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:21:50.542925 10768 raft_consensus.cc:729] T 00000000000000000000000000000000 P a7373ecfc6ca4b93b4f91158b42ad354 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7373ecfc6ca4b93b4f91158b42ad354, State: Initialized, Role: FOLLOWER
I20260929 03:21:50.543074 10768 consensus_queue.cc:260] T 00000000000000000000000000000000 P a7373ecfc6ca4b93b4f91158b42ad354 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7373ecfc6ca4b93b4f91158b42ad354" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 38457 } }
I20260929 03:21:50.543176 10768 raft_consensus.cc:388] T 00000000000000000000000000000000 P a7373ecfc6ca4b93b4f91158b42ad354 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:21:50.543221 10768 raft_consensus.cc:482] T 00000000000000000000000000000000 P a7373ecfc6ca4b93b4f91158b42ad354 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:21:50.543273 10768 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a7373ecfc6ca4b93b4f91158b42ad354 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:21:50.543792 10768 raft_consensus.cc:504] T 00000000000000000000000000000000 P a7373ecfc6ca4b93b4f91158b42ad354 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7373ecfc6ca4b93b4f91158b42ad354" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 38457 } }
I20260929 03:21:50.543903 10768 leader_election.cc:302] T 00000000000000000000000000000000 P a7373ecfc6ca4b93b4f91158b42ad354 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a7373ecfc6ca4b93b4f91158b42ad354; no voters:
I20260929 03:21:50.544075 10768 leader_election.cc:288] T 00000000000000000000000000000000 P a7373ecfc6ca4b93b4f91158b42ad354 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:21:50.544108 10773 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a7373ecfc6ca4b93b4f91158b42ad354 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:21:50.544210 10773 raft_consensus.cc:686] T 00000000000000000000000000000000 P a7373ecfc6ca4b93b4f91158b42ad354 [term 1 LEADER]: Becoming Leader. State: Replica: a7373ecfc6ca4b93b4f91158b42ad354, State: Running, Role: LEADER
I20260929 03:21:50.544340 10768 sys_catalog.cc:564] T 00000000000000000000000000000000 P a7373ecfc6ca4b93b4f91158b42ad354 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:21:50.544344 10773 consensus_queue.cc:237] T 00000000000000000000000000000000 P a7373ecfc6ca4b93b4f91158b42ad354 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7373ecfc6ca4b93b4f91158b42ad354" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 38457 } }
I20260929 03:21:50.544735 10773 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7373ecfc6ca4b93b4f91158b42ad354 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a7373ecfc6ca4b93b4f91158b42ad354. Latest consensus state: current_term: 1 leader_uuid: "a7373ecfc6ca4b93b4f91158b42ad354" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7373ecfc6ca4b93b4f91158b42ad354" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 38457 } } }
I20260929 03:21:50.544745 10774 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7373ecfc6ca4b93b4f91158b42ad354 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a7373ecfc6ca4b93b4f91158b42ad354" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7373ecfc6ca4b93b4f91158b42ad354" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 38457 } } }
I20260929 03:21:50.544901 10774 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7373ecfc6ca4b93b4f91158b42ad354 [sys.catalog]: This master's current role is: LEADER
I20260929 03:21:50.545099 10773 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7373ecfc6ca4b93b4f91158b42ad354 [sys.catalog]: This master's current role is: LEADER
W20260929 03:21:50.546243 10789 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a7373ecfc6ca4b93b4f91158b42ad354: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 03:21:50.546341 10789 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 03:21:50.546406 10780 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:21:50.546563 10780 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:21:50.547753 10772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:50.541634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49881 (local address 127.20.178.126:38457)
0929 03:21:50.541809 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0929 03:21:50.541817 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 03:21:50.541841 (+ 24us) server_negotiation.cc:407] Connection header received
0929 03:21:50.542693 (+ 852us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:21:50.542699 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:21:50.542745 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:21:50.542853 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:21:50.543100 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:50.544500 (+ 1400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:50.545604 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:50.546110 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:50.546231 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:21:50.546916 (+ 685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:21:50.546920 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:21:50.546920 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:21:50.547139 (+ 219us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:21:50.547177 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:21:50.547191 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:21:50.547256 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0929 03:21:50.547365 (+ 109us) server_negotiation.cc:299] Negotiation successful
0929 03:21:50.547549 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":83,"threads_started":1}
I20260929 03:21:50.548077 10780 catalog_manager.cc:1347] Generated new cluster ID: 285b1c9c2c674b36ad8f1a8c07f9f6fc
I20260929 03:21:50.548126 10780 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:21:50.548517 21193 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 03:21:50.559841 10772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:50.556966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58211 (local address 127.20.178.126:38457)
0929 03:21:50.557017 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0929 03:21:50.557019 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:21:50.557140 (+ 121us) server_negotiation.cc:407] Connection header received
0929 03:21:50.557206 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:21:50.557207 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:21:50.557222 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:21:50.557295 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:21:50.557461 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:50.557936 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:50.558634 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:50.558823 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:50.558909 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:21:50.559559 (+ 650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:21:50.559561 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:21:50.559561 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:21:50.559565 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:21:50.559580 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:21:50.559584 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:21:50.559649 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0929 03:21:50.559710 (+ 61us) server_negotiation.cc:299] Negotiation successful
0929 03:21:50.559733 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260929 03:21:50.576076 10780 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:21:50.576879 10780 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:21:50.590380 10780 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a7373ecfc6ca4b93b4f91158b42ad354: Generated new TSK 0
I20260929 03:21:50.590593 10780 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 03:21:50.595937 21193 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuoaHbX/build/release/bin/kudu with pid 10700
2026-09-29T03:21:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1339 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790651852476208-21193-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:21:50 dist-test-slave-c9qh krb5kdc[10799](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:21:50 dist-test-slave-c9qh krb5kdc[10799](info): set up 2 sockets
Sep 29 03:21:50 dist-test-slave-c9qh krb5kdc[10799](info): commencing operation
krb5kdc: starting...
W20260929 03:21:51.620747 21193 mini_kdc.cc:121] Time spent starting KDC: real 1.007s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790651852476208-21193-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790651852476208-21193-0/krb5kdc/test-user.keytab.
Sep 29 03:21:51 dist-test-slave-c9qh krb5kdc[10799](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652111, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:21:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:21:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.178.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.178.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.178.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.126.keytab.
Entry for principal kudu/127.20.178.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.126.keytab.
I20260929 03:21:51.733328 21193 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuoaHbX/build/release/bin/kudu
/tmp/dist-test-taskuoaHbX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790651852476208-21193-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790651852476208-21193-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790651852476208-21193-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790651852476208-21193-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.178.126:38087
--webserver_interface=127.20.178.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.126.keytab
--principal=kudu/127.20.178.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.178.84:39705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.178.126:38087
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790651852476208-21193-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790651852476208-21193-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790651852476208-21193-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:21:51.817638 10812 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:21:51.817854 10812 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:21:51.817878 10812 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:21:51.819068 10812 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:21:51.819111 10812 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:21:51.819126 10812 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:21:51.819164 10812 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:21:51.819195 10812 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:21:51.820678 10812 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.178.84:39705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790651852476208-21193-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790651852476208-21193-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.178.126:38087
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.126.keytab
--principal=kudu/127.20.178.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.178.126:38087
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790651852476208-21193-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.178.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10812
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790651852476208-21193-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:21:51.820932 10812 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:21:51.821156 10812 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:21:51.826022 10817 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:21:51.826002 10818 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:21:51.826002 10820 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:21:51.826398 10812 server_base.cc:1034] running on GCE node
I20260929 03:21:51.826635 10812 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:21:51.826910 10812 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:21:51.828055 10812 hybrid_clock.cc:648] HybridClock initialized: now 1790652111828020 us; error 51 us; skew 500 ppm
Sep 29 03:21:51 dist-test-slave-c9qh krb5kdc[10799](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652111, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:21:51.830502 10812 init.cc:377] Logged in from keytab as kudu/127.20.178.126@KRBTEST.COM (short username kudu)
I20260929 03:21:51.830935 10812 webserver.cc:460] Webserver started at http://127.20.178.126:42577/ using document root <none> and password file <none>
I20260929 03:21:51.831115 10812 fs_manager.cc:359] Metadata directory not provided
I20260929 03:21:51.831153 10812 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:21:51.831239 10812 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:21:51.832222 10812 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790651852476208-21193-0/minicluster-data/master-0/data/instance:
uuid: "5144f391722444c6a08af245e274414e"
format_stamp: "Formatted at 2026-09-29 03:21:51 on dist-test-slave-c9qh"
server_key: "6165c30a08350f83f2546752231c61f5"
server_key_iv: "4700e66f07bb49f70b52234bf6c4dc65"
server_key_version: "encryptionkey@0"
I20260929 03:21:51.832521 10812 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790651852476208-21193-0/minicluster-data/master-0/wal/instance:
uuid: "5144f391722444c6a08af245e274414e"
format_stamp: "Formatted at 2026-09-29 03:21:51 on dist-test-slave-c9qh"
server_key: "6165c30a08350f83f2546752231c61f5"
server_key_iv: "4700e66f07bb49f70b52234bf6c4dc65"
server_key_version: "encryptionkey@0"
I20260929 03:21:51.833855 10812 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260929 03:21:51.834601 10827 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:21:51.834781 10812 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260929 03:21:51.834882 10812 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790651852476208-21193-0/minicluster-data/master-0/data,/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790651852476208-21193-0/minicluster-data/master-0/wal
uuid: "5144f391722444c6a08af245e274414e"
format_stamp: "Formatted at 2026-09-29 03:21:51 on dist-test-slave-c9qh"
server_key: "6165c30a08350f83f2546752231c61f5"
server_key_iv: "4700e66f07bb49f70b52234bf6c4dc65"
server_key_version: "encryptionkey@0"
I20260929 03:21:51.834965 10812 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790651852476208-21193-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790651852476208-21193-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790651852476208-21193-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:21:51.838315 10812 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:21:51.840556 10812 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:21:51.840751 10812 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:21:51.844702 10812 rpc_server.cc:307] RPC server started. Bound to: 127.20.178.126:38087
I20260929 03:21:51.844735 10879 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.178.126:38087 every 8 connection(s)
I20260929 03:21:51.845072 10812 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790651852476208-21193-0/minicluster-data/master-0/data/info.pb
I20260929 03:21:51.846022 10880 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:21:51.847616 21193 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuoaHbX/build/release/bin/kudu as pid 10812
I20260929 03:21:51.847711 21193 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790651852476208-21193-0/minicluster-data/master-0/wal/instance
I20260929 03:21:51.847906 21193 external_mini_cluster.cc:1428] Setting key 4b4fe920221f25a9d87e4d7809364bdf
I20260929 03:21:51.848723 10880 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5144f391722444c6a08af245e274414e: Bootstrap starting.
I20260929 03:21:51.849340 10880 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5144f391722444c6a08af245e274414e: Neither blocks nor log segments found. Creating new log.
I20260929 03:21:51.849663 10880 log.cc:824] T 00000000000000000000000000000000 P 5144f391722444c6a08af245e274414e: Log is configured to *not* fsync() on all Append() calls
I20260929 03:21:51.850366 10883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:51.848443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60779 (local address 127.20.178.126:38087)
0929 03:21:51.848608 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0929 03:21:51.848616 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 03:21:51.848678 (+ 62us) server_negotiation.cc:407] Connection header received
0929 03:21:51.849149 (+ 471us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:21:51.849155 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:21:51.849181 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:21:51.849296 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:21:51.849949 (+ 653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:21:51.849953 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:21:51.849953 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:21:51.849958 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:21:51.849999 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:21:51.850014 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:21:51.850109 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0929 03:21:51.850142 (+ 33us) server_negotiation.cc:299] Negotiation successful
0929 03:21:51.850182 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":74,"threads_started":1}
I20260929 03:21:51.850471 10880 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5144f391722444c6a08af245e274414e: No bootstrap required, opened a new log
I20260929 03:21:51.851329 21193 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 03:21:51.853084 10880 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5144f391722444c6a08af245e274414e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5144f391722444c6a08af245e274414e" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 38087 } }
I20260929 03:21:51.853281 10880 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5144f391722444c6a08af245e274414e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:21:51.853370 10880 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5144f391722444c6a08af245e274414e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5144f391722444c6a08af245e274414e, State: Initialized, Role: FOLLOWER
I20260929 03:21:51.853529 10880 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5144f391722444c6a08af245e274414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5144f391722444c6a08af245e274414e" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 38087 } }
I20260929 03:21:51.853634 10880 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5144f391722444c6a08af245e274414e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:21:51.853679 10880 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5144f391722444c6a08af245e274414e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:21:51.853740 10880 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5144f391722444c6a08af245e274414e [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:21:51.853974 10883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:51.852960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55109 (local address 127.20.178.126:38087)
0929 03:21:51.852998 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0929 03:21:51.853000 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:21:51.853055 (+ 55us) server_negotiation.cc:407] Connection header received
0929 03:21:51.853120 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:21:51.853121 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:21:51.853124 (+ 3us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:21:51.853167 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:21:51.853729 (+ 562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:21:51.853730 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:21:51.853731 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:21:51.853735 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:21:51.853751 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:21:51.853755 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:21:51.853810 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0929 03:21:51.853870 (+ 60us) server_negotiation.cc:299] Negotiation successful
0929 03:21:51.853881 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260929 03:21:51.854449 10880 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5144f391722444c6a08af245e274414e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5144f391722444c6a08af245e274414e" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 38087 } }
I20260929 03:21:51.854534 10880 leader_election.cc:302] T 00000000000000000000000000000000 P 5144f391722444c6a08af245e274414e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5144f391722444c6a08af245e274414e; no voters:
I20260929 03:21:51.854648 10880 leader_election.cc:288] T 00000000000000000000000000000000 P 5144f391722444c6a08af245e274414e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:21:51.854697 10890 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5144f391722444c6a08af245e274414e [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:21:51.854837 10890 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5144f391722444c6a08af245e274414e [term 1 LEADER]: Becoming Leader. State: Replica: 5144f391722444c6a08af245e274414e, State: Running, Role: LEADER
I20260929 03:21:51.854864 10880 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5144f391722444c6a08af245e274414e [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:21:51.854954 10890 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5144f391722444c6a08af245e274414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5144f391722444c6a08af245e274414e" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 38087 } }
I20260929 03:21:51.855332 10891 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5144f391722444c6a08af245e274414e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5144f391722444c6a08af245e274414e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5144f391722444c6a08af245e274414e" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 38087 } } }
I20260929 03:21:51.855340 10892 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5144f391722444c6a08af245e274414e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5144f391722444c6a08af245e274414e. Latest consensus state: current_term: 1 leader_uuid: "5144f391722444c6a08af245e274414e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5144f391722444c6a08af245e274414e" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 38087 } } }
I20260929 03:21:51.855518 10891 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5144f391722444c6a08af245e274414e [sys.catalog]: This master's current role is: LEADER
I20260929 03:21:51.855535 10892 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5144f391722444c6a08af245e274414e [sys.catalog]: This master's current role is: LEADER
W20260929 03:21:51.856619 10906 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5144f391722444c6a08af245e274414e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 03:21:51.856709 10906 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 03:21:51.856752 10899 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:21:51.856892 10899 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:21:51.858305 10899 catalog_manager.cc:1347] Generated new cluster ID: 18187497906242c8b7c4e4b754af4acd
I20260929 03:21:51.858354 10899 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:21:51.872946 10899 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:21:51.873143 10899 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:21:51.883412 10899 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5144f391722444c6a08af245e274414e: Generated new TSK 0
I20260929 03:21:51.883656 10899 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 03:21:51.893241 21193 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuoaHbX/build/release/bin/kudu with pid 10812
2026-09-29T03:21:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1297 ms)
[----------] 18 tests from AuthTokenIssuingTest (23708 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651852476208-21193-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:21:51 dist-test-slave-c9qh krb5kdc[10911](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:21:51 dist-test-slave-c9qh krb5kdc[10911](info): set up 2 sockets
Sep 29 03:21:51 dist-test-slave-c9qh krb5kdc[10911](info): commencing operation
krb5kdc: starting...
W20260929 03:21:52.916879 21193 mini_kdc.cc:121] Time spent starting KDC: real 1.006s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651852476208-21193-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651852476208-21193-0/krb5kdc/test-user.keytab.
Sep 29 03:21:52 dist-test-slave-c9qh krb5kdc[10911](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652112, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:21:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:21:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.178.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.178.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.178.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.126.keytab.
Entry for principal kudu/127.20.178.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.126.keytab.
I20260929 03:21:53.031247 21193 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuoaHbX/build/release/bin/kudu
/tmp/dist-test-taskuoaHbX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651852476208-21193-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651852476208-21193-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651852476208-21193-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651852476208-21193-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.178.126:35917
--webserver_interface=127.20.178.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.126.keytab
--principal=kudu/127.20.178.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.178.84:45961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.178.126:35917,127.20.178.125:33321,127.20.178.124:46467
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651852476208-21193-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651852476208-21193-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651852476208-21193-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:21:53.109428 10924 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 03:21:53.109678 10924 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:21:53.109719 10924 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:21:53.109761 10924 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:21:53.110904 10924 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:21:53.110962 10924 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:21:53.110987 10924 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:21:53.111011 10924 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:21:53.111035 10924 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:21:53.112438 10924 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.178.84:45961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651852476208-21193-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651852476208-21193-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.178.126:35917,127.20.178.125:33321,127.20.178.124:46467
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.126.keytab
--principal=kudu/127.20.178.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.178.126:35917
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651852476208-21193-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.178.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10924
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651852476208-21193-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:21:53.112730 10924 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:21:53.112931 10924 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:21:53.117447 10930 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:21:53.117487 10932 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:21:53.117703 10924 server_base.cc:1034] running on GCE node
W20260929 03:21:53.117479 10929 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:21:53.117990 10924 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:21:53.118268 10924 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:21:53.119400 10924 hybrid_clock.cc:648] HybridClock initialized: now 1790652113119379 us; error 33 us; skew 500 ppm
Sep 29 03:21:53 dist-test-slave-c9qh krb5kdc[10911](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652113, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:21:53.121752 10924 init.cc:377] Logged in from keytab as kudu/127.20.178.126@KRBTEST.COM (short username kudu)
I20260929 03:21:53.122174 10924 webserver.cc:460] Webserver started at http://127.20.178.126:39365/ using document root <none> and password file <none>
I20260929 03:21:53.122375 10924 fs_manager.cc:359] Metadata directory not provided
I20260929 03:21:53.122416 10924 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:21:53.122571 10924 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:21:53.123554 10924 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651852476208-21193-0/minicluster-data/master-0/data/instance:
uuid: "57f8519c269e456dbab3c7fa4665d876"
format_stamp: "Formatted at 2026-09-29 03:21:53 on dist-test-slave-c9qh"
server_key: "4a0aff48eca8b6035d58c42b9eb4abf8"
server_key_iv: "c43797cf72512e388de5b02e570815bf"
server_key_version: "encryptionkey@0"
I20260929 03:21:53.123888 10924 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651852476208-21193-0/minicluster-data/master-0/wal/instance:
uuid: "57f8519c269e456dbab3c7fa4665d876"
format_stamp: "Formatted at 2026-09-29 03:21:53 on dist-test-slave-c9qh"
server_key: "4a0aff48eca8b6035d58c42b9eb4abf8"
server_key_iv: "c43797cf72512e388de5b02e570815bf"
server_key_version: "encryptionkey@0"
I20260929 03:21:53.125227 10924 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 03:21:53.125888 10939 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:21:53.126075 10924 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 03:21:53.126159 10924 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651852476208-21193-0/minicluster-data/master-0/data,/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651852476208-21193-0/minicluster-data/master-0/wal
uuid: "57f8519c269e456dbab3c7fa4665d876"
format_stamp: "Formatted at 2026-09-29 03:21:53 on dist-test-slave-c9qh"
server_key: "4a0aff48eca8b6035d58c42b9eb4abf8"
server_key_iv: "c43797cf72512e388de5b02e570815bf"
server_key_version: "encryptionkey@0"
I20260929 03:21:53.126224 10924 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651852476208-21193-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651852476208-21193-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651852476208-21193-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:21:53.134536 10924 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:21:53.136586 10924 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:21:53.136775 10924 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:21:53.140743 10924 rpc_server.cc:307] RPC server started. Bound to: 127.20.178.126:35917
I20260929 03:21:53.140801 10991 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.178.126:35917 every 8 connection(s)
I20260929 03:21:53.141084 10924 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651852476208-21193-0/minicluster-data/master-0/data/info.pb
I20260929 03:21:53.141840 10992 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:21:53.143301 10992 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.178.126" port: 35917 } has no permanent_uuid. Determining permanent_uuid...
I20260929 03:21:53.145117 21193 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuoaHbX/build/release/bin/kudu as pid 10924
I20260929 03:21:53.145196 21193 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651852476208-21193-0/minicluster-data/master-0/wal/instance
I20260929 03:21:53.145388 21193 external_mini_cluster.cc:1428] Setting key 6020d562c6829c297772ee01b49e81d2
Sep 29 03:21:53 dist-test-slave-c9qh krb5kdc[10911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652113, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.126@KRBTEST.COM for kudu/127.20.178.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.20.178.125@KRBTEST.COM; defaulting to no policy
I20260929 03:21:53.153287 10994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:53.143883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:35917 (local address 127.0.0.1:47766)
0929 03:21:53.144044 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0929 03:21:53.144074 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0929 03:21:53.144491 (+ 417us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:21:53.144846 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:21:53.144849 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:21:53.145070 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:21:53.145282 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:21:53.145287 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:21:53.146424 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:21:53.146425 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:21:53.147396 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:21:53.147398 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:21:53.147878 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:21:53.147879 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:21:53.147939 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:21:53.148416 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:21:53.148426 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:21:53.150217 (+ 1791us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:21:53.151840 (+ 1623us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:21:53.151848 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:21:53.151850 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:21:53.152126 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:21:53.152347 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:21:53.152348 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:21:53.152349 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:21:53.152466 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:21:53.152877 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:21:53.152879 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:21:53.152922 (+ 43us) client_negotiation.cc:769] Sending connection context
0929 03:21:53.152995 (+ 73us) client_negotiation.cc:240] Negotiation successful
0929 03:21:53.153107 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":80,"threads_started":1}
I20260929 03:21:53.153290 10995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:53.143906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47766 (local address 127.20.178.126:35917)
0929 03:21:53.144613 (+ 707us) server_negotiation.cc:206] Beginning negotiation
0929 03:21:53.144617 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:21:53.144627 (+ 10us) server_negotiation.cc:407] Connection header received
0929 03:21:53.144739 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:21:53.144740 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:21:53.144755 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:21:53.144819 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:21:53.145378 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:53.146331 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:53.147475 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:53.147788 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:53.147906 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:21:53.150304 (+ 2398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:21:53.150310 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:21:53.150310 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:21:53.150316 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:21:53.151759 (+ 1443us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:21:53.152208 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:21:53.152211 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:21:53.152211 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:21:53.152280 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:21:53.152530 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:21:53.152530 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:21:53.152531 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:21:53.152766 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:21:53.152858 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0929 03:21:53.153055 (+ 197us) server_negotiation.cc:299] Negotiation successful
0929 03:21:53.153175 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":36,"server-negotiator.queue_time_us":688,"thread_start_us":91,"threads_started":1}
I20260929 03:21:53.154192 10992 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.178.125" port: 33321 } has no permanent_uuid. Determining permanent_uuid...
I20260929 03:21:53.154507 10994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:53.154411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.125:33321 (local address 127.0.0.1:54306)
0929 03:21:53.154450 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0929 03:21:53.154491 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.178.125:33321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260929 03:21:53.154644 10940 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.178.125:33321: connect: Connection refused (error 111)
I20260929 03:21:53.155028 10994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:53.154935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.125:33321 (local address 127.0.0.1:54308)
0929 03:21:53.154990 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0929 03:21:53.155014 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.178.125:33321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260929 03:21:53.155119 10992 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.178.125:33321: Network error: Client connection negotiation failed: client connection to 127.20.178.125:33321: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.178.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.178.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.125.keytab.
Entry for principal kudu/127.20.178.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.125.keytab.
I20260929 03:21:53.169493 21193 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuoaHbX/build/release/bin/kudu
/tmp/dist-test-taskuoaHbX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651852476208-21193-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651852476208-21193-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651852476208-21193-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651852476208-21193-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.178.125:33321
--webserver_interface=127.20.178.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.125.keytab
--principal=kudu/127.20.178.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.178.84:45961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.178.126:35917,127.20.178.125:33321,127.20.178.124:46467
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651852476208-21193-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651852476208-21193-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651852476208-21193-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 03:21:53.204329 10992 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.178.125" port: 33321 } attempt: 1
I20260929 03:21:53.204926 10994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:53.204764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.125:33321 (local address 127.0.0.1:54322)
0929 03:21:53.204841 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0929 03:21:53.204889 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.178.125:33321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260929 03:21:53.205323 10994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:53.205224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.125:33321 (local address 127.0.0.1:54330)
0929 03:21:53.205270 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0929 03:21:53.205299 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.178.125:33321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260929 03:21:53.205417 10992 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.178.125:33321: Network error: Client connection negotiation failed: client connection to 127.20.178.125:33321: connect: Connection refused (error 111)
W20260929 03:21:53.248869 10999 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 03:21:53.249060 10999 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:21:53.249078 10999 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:21:53.249096 10999 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:21:53.250162 10999 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:21:53.250197 10999 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:21:53.250211 10999 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:21:53.250223 10999 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:21:53.250236 10999 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:21:53.251578 10999 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.178.84:45961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651852476208-21193-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651852476208-21193-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.20.178.126:35917,127.20.178.125:33321,127.20.178.124:46467
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.125.keytab
--principal=kudu/127.20.178.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.178.125:33321
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651852476208-21193-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.20.178.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10999
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651852476208-21193-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:21:53.251750 10999 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:21:53.251911 10999 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:21:53.256421 11007 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:21:53.256433 11004 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:21:53.256587 10999 server_base.cc:1034] running on GCE node
W20260929 03:21:53.256433 11005 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:21:53.256942 10999 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:21:53.257180 10999 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:21:53.258328 10999 hybrid_clock.cc:648] HybridClock initialized: now 1790652113258309 us; error 35 us; skew 500 ppm
Sep 29 03:21:53 dist-test-slave-c9qh krb5kdc[10911](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652113, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:21:53.260797 10999 init.cc:377] Logged in from keytab as kudu/127.20.178.125@KRBTEST.COM (short username kudu)
I20260929 03:21:53.261250 10999 webserver.cc:460] Webserver started at http://127.20.178.125:32957/ using document root <none> and password file <none>
I20260929 03:21:53.261475 10999 fs_manager.cc:359] Metadata directory not provided
I20260929 03:21:53.261519 10999 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:21:53.261664 10999 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:21:53.262686 10999 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651852476208-21193-0/minicluster-data/master-1/data/instance:
uuid: "dda8ab636c9047e28e7d528c7987af6b"
format_stamp: "Formatted at 2026-09-29 03:21:53 on dist-test-slave-c9qh"
server_key: "52614f9237f5b4c6ee97be93268ee549"
server_key_iv: "40afa475de20c43dfe3cc73fa1a5f355"
server_key_version: "encryptionkey@0"
I20260929 03:21:53.263039 10999 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651852476208-21193-0/minicluster-data/master-1/wal/instance:
uuid: "dda8ab636c9047e28e7d528c7987af6b"
format_stamp: "Formatted at 2026-09-29 03:21:53 on dist-test-slave-c9qh"
server_key: "52614f9237f5b4c6ee97be93268ee549"
server_key_iv: "40afa475de20c43dfe3cc73fa1a5f355"
server_key_version: "encryptionkey@0"
I20260929 03:21:53.264353 10999 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 03:21:53.265125 11014 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:21:53.265287 10999 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260929 03:21:53.265385 10999 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651852476208-21193-0/minicluster-data/master-1/data,/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651852476208-21193-0/minicluster-data/master-1/wal
uuid: "dda8ab636c9047e28e7d528c7987af6b"
format_stamp: "Formatted at 2026-09-29 03:21:53 on dist-test-slave-c9qh"
server_key: "52614f9237f5b4c6ee97be93268ee549"
server_key_iv: "40afa475de20c43dfe3cc73fa1a5f355"
server_key_version: "encryptionkey@0"
I20260929 03:21:53.265467 10999 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651852476208-21193-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651852476208-21193-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651852476208-21193-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:21:53.273530 10992 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.178.125" port: 33321 } attempt: 2
I20260929 03:21:53.273881 10994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:53.273767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.125:33321 (local address 127.0.0.1:54340)
0929 03:21:53.273819 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0929 03:21:53.273853 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.178.125:33321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260929 03:21:53.274169 10994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:53.274096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.125:33321 (local address 127.0.0.1:54356)
0929 03:21:53.274131 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0929 03:21:53.274154 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.178.125:33321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260929 03:21:53.274259 10992 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.178.125:33321: Network error: Client connection negotiation failed: client connection to 127.20.178.125:33321: connect: Connection refused (error 111)
I20260929 03:21:53.287403 10999 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:21:53.289639 10999 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:21:53.289798 10999 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:21:53.294135 10999 rpc_server.cc:307] RPC server started. Bound to: 127.20.178.125:33321
I20260929 03:21:53.294191 11066 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.178.125:33321 every 8 connection(s)
I20260929 03:21:53.294569 10999 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651852476208-21193-0/minicluster-data/master-1/data/info.pb
I20260929 03:21:53.295382 11067 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:21:53.296922 11067 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.178.126" port: 35917 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 03:21:53 dist-test-slave-c9qh krb5kdc[10911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652113, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.125@KRBTEST.COM for kudu/127.20.178.126@KRBTEST.COM
I20260929 03:21:53.303951 21193 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuoaHbX/build/release/bin/kudu as pid 10999
I20260929 03:21:53.304111 21193 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651852476208-21193-0/minicluster-data/master-1/wal/instance
I20260929 03:21:53.304410 21193 external_mini_cluster.cc:1428] Setting key 784b65b81ddf9eecc4bd94b90ca4cf63
I20260929 03:21:53.305476 11069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:53.297508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:35917 (local address 127.0.0.1:47774)
0929 03:21:53.297660 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0929 03:21:53.297691 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0929 03:21:53.298120 (+ 429us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:21:53.298306 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:21:53.298309 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:21:53.298546 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:21:53.298724 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:21:53.298727 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:21:53.299315 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:21:53.299316 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:21:53.300102 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:21:53.300104 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:21:53.300432 (+ 328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:21:53.300433 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:21:53.300540 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:21:53.301017 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:21:53.301025 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:21:53.302296 (+ 1271us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:21:53.303852 (+ 1556us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:21:53.303858 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:21:53.303860 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:21:53.304125 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:21:53.304525 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:21:53.304526 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:21:53.304527 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:21:53.304696 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:21:53.304979 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:21:53.304984 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:21:53.305126 (+ 142us) client_negotiation.cc:769] Sending connection context
0929 03:21:53.305210 (+ 84us) client_negotiation.cc:240] Negotiation successful
0929 03:21:53.305326 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":89,"threads_started":1}
I20260929 03:21:53.305488 10995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:53.297540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47774 (local address 127.20.178.126:35917)
0929 03:21:53.297579 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0929 03:21:53.297581 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:21:53.297768 (+ 187us) server_negotiation.cc:407] Connection header received
0929 03:21:53.298197 (+ 429us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:21:53.298198 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:21:53.298210 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:21:53.298259 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:21:53.298788 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:53.299245 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:53.300171 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:53.300347 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:53.300397 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:21:53.302420 (+ 2023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:21:53.302422 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:21:53.302422 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:21:53.302427 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:21:53.303763 (+ 1336us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:21:53.304350 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:21:53.304351 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:21:53.304352 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:21:53.304440 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:21:53.304768 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:21:53.304769 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:21:53.304769 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:21:53.304914 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:21:53.304990 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0929 03:21:53.305248 (+ 258us) server_negotiation.cc:299] Negotiation successful
0929 03:21:53.305340 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260929 03:21:53.306042 11067 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.178.125" port: 33321 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 03:21:53 dist-test-slave-c9qh krb5kdc[10911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652113, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.125@KRBTEST.COM for kudu/127.20.178.125@KRBTEST.COM
WARNING: no policy specified for kudu/127.20.178.124@KRBTEST.COM; defaulting to no policy
I20260929 03:21:53.313365 11069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:53.306289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.125:33321 (local address 127.0.0.1:54372)
0929 03:21:53.306347 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0929 03:21:53.306356 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 03:21:53.306396 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:21:53.306633 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:21:53.306634 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:21:53.306768 (+ 134us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:21:53.306808 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:21:53.306810 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:21:53.307641 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:21:53.307642 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:21:53.308271 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:21:53.308272 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:21:53.308820 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:21:53.308821 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:21:53.308882 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:21:53.309291 (+ 409us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:21:53.309295 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:21:53.310644 (+ 1349us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:21:53.312351 (+ 1707us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:21:53.312352 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:21:53.312353 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:21:53.312513 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:21:53.312731 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:21:53.312732 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:21:53.312733 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:21:53.312782 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:21:53.313052 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:21:53.313052 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:21:53.313089 (+ 37us) client_negotiation.cc:769] Sending connection context
0929 03:21:53.313148 (+ 59us) client_negotiation.cc:240] Negotiation successful
0929 03:21:53.313242 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260929 03:21:53.313429 11071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:53.306354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54372 (local address 127.20.178.125:33321)
0929 03:21:53.306513 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0929 03:21:53.306517 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:21:53.306531 (+ 14us) server_negotiation.cc:407] Connection header received
0929 03:21:53.306547 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:21:53.306548 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:21:53.306557 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:21:53.306632 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:21:53.306885 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:53.307546 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:53.308360 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:53.308738 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:53.308821 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:21:53.310733 (+ 1912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:21:53.310740 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:21:53.310741 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:21:53.310747 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:21:53.312267 (+ 1520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:21:53.312602 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:21:53.312605 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:21:53.312605 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:21:53.312676 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:21:53.312848 (+ 172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:21:53.312849 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:21:53.312849 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:21:53.312983 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:21:53.313071 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0929 03:21:53.313192 (+ 121us) server_negotiation.cc:299] Negotiation successful
0929 03:21:53.313303 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":35,"server-negotiator.queue_time_us":126,"thread_start_us":88,"threads_started":1}
I20260929 03:21:53.314111 11067 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.178.124" port: 46467 } has no permanent_uuid. Determining permanent_uuid...
I20260929 03:21:53.314427 11069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:53.314326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.124:46467 (local address 127.0.0.1:43134)
0929 03:21:53.314367 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0929 03:21:53.314408 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.178.124:46467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.178.124@KRBTEST.COM" created.
W20260929 03:21:53.314632 11017 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.178.124:46467: connect: Connection refused (error 111)
I20260929 03:21:53.315030 11069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:53.314939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.124:46467 (local address 127.0.0.1:43142)
0929 03:21:53.314983 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0929 03:21:53.315007 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.178.124:46467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260929 03:21:53.315129 11067 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.178.124:46467: Network error: Client connection negotiation failed: client connection to 127.20.178.124:46467: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.178.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.124.keytab.
Entry for principal kudu/127.20.178.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.124.keytab.
I20260929 03:21:53.327147 21193 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuoaHbX/build/release/bin/kudu
/tmp/dist-test-taskuoaHbX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651852476208-21193-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651852476208-21193-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651852476208-21193-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651852476208-21193-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.178.124:46467
--webserver_interface=127.20.178.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.124.keytab
--principal=kudu/127.20.178.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.178.84:45961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.178.126:35917,127.20.178.125:33321,127.20.178.124:46467
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651852476208-21193-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651852476208-21193-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651852476208-21193-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 03:21:53.364354 11067 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.178.124" port: 46467 } attempt: 1
I20260929 03:21:53.364938 11069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:53.364799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.124:46467 (local address 127.0.0.1:43150)
0929 03:21:53.364857 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0929 03:21:53.364902 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.178.124:46467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260929 03:21:53.365280 11069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:53.365195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.124:46467 (local address 127.0.0.1:43156)
0929 03:21:53.365236 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0929 03:21:53.365261 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.178.124:46467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260929 03:21:53.365377 10992 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.178.125" port: 33321 } attempt: 3
W20260929 03:21:53.365398 11067 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.178.124:46467: Network error: Client connection negotiation failed: client connection to 127.20.178.124:46467: connect: Connection refused (error 111)
Sep 29 03:21:53 dist-test-slave-c9qh krb5kdc[10911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652113, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.126@KRBTEST.COM for kudu/127.20.178.125@KRBTEST.COM
I20260929 03:21:53.372443 10994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:53.365608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.125:33321 (local address 127.0.0.1:54386)
0929 03:21:53.365657 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0929 03:21:53.365668 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 03:21:53.365726 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:21:53.365850 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:21:53.365851 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:21:53.366024 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:21:53.366097 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:21:53.366098 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:21:53.366693 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:21:53.366694 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:21:53.367488 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:21:53.367491 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:21:53.367932 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:21:53.367933 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:21:53.368009 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:21:53.368419 (+ 410us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:21:53.368426 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:21:53.369911 (+ 1485us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:21:53.371588 (+ 1677us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:21:53.371588 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:21:53.371589 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:21:53.371745 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:21:53.371943 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:21:53.371944 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:21:53.371944 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:21:53.371997 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:21:53.372237 (+ 240us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:21:53.372238 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:21:53.372272 (+ 34us) client_negotiation.cc:769] Sending connection context
0929 03:21:53.372318 (+ 46us) client_negotiation.cc:240] Negotiation successful
0929 03:21:53.372382 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260929 03:21:53.372509 11071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:53.365653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54386 (local address 127.20.178.125:33321)
0929 03:21:53.365692 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0929 03:21:53.365694 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:21:53.365709 (+ 15us) server_negotiation.cc:407] Connection header received
0929 03:21:53.365786 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:21:53.365787 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:21:53.365796 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:21:53.365836 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:21:53.366162 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:53.366624 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:53.367568 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:53.367865 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:53.367917 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:21:53.369992 (+ 2075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:21:53.369994 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:21:53.369994 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:21:53.370002 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:21:53.371507 (+ 1505us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:21:53.371803 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:21:53.371805 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:21:53.371805 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:21:53.371883 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:21:53.372051 (+ 168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:21:53.372052 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:21:53.372052 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:21:53.372174 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:21:53.372233 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0929 03:21:53.372350 (+ 117us) server_negotiation.cc:299] Negotiation successful
0929 03:21:53.372422 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260929 03:21:53.373131 10992 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.178.124" port: 46467 } has no permanent_uuid. Determining permanent_uuid...
I20260929 03:21:53.373437 10994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:53.373343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.124:46467 (local address 127.0.0.1:43164)
0929 03:21:53.373385 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0929 03:21:53.373412 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.178.124:46467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260929 03:21:53.373781 10994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:53.373709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.124:46467 (local address 127.0.0.1:43178)
0929 03:21:53.373746 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0929 03:21:53.373767 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.178.124:46467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260929 03:21:53.373878 10992 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.178.124:46467: Network error: Client connection negotiation failed: client connection to 127.20.178.124:46467: connect: Connection refused (error 111)
I20260929 03:21:53.405030 10992 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.178.124" port: 46467 } attempt: 1
I20260929 03:21:53.405516 10994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:53.405381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.124:46467 (local address 127.0.0.1:43182)
0929 03:21:53.405441 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0929 03:21:53.405489 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.178.124:46467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260929 03:21:53.405900 10994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:53.405818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.124:46467 (local address 127.0.0.1:43198)
0929 03:21:53.405862 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0929 03:21:53.405884 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.178.124:46467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260929 03:21:53.406016 10992 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.178.124:46467: Network error: Client connection negotiation failed: client connection to 127.20.178.124:46467: connect: Connection refused (error 111)
W20260929 03:21:53.407811 11074 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 03:21:53.408064 11074 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:21:53.408102 11074 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:21:53.408145 11074 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:21:53.409322 11074 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:21:53.409359 11074 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:21:53.409374 11074 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:21:53.409412 11074 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:21:53.409425 11074 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:21:53.410807 11074 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.178.84:45961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651852476208-21193-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651852476208-21193-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.20.178.126:35917,127.20.178.125:33321,127.20.178.124:46467
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.124.keytab
--principal=kudu/127.20.178.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.178.124:46467
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651852476208-21193-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.20.178.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11074
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651852476208-21193-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:21:53.410981 11074 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:21:53.411140 11074 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:21:53.415692 11080 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:21:53.415706 11074 server_base.cc:1034] running on GCE node
W20260929 03:21:53.415796 11079 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:21:53.415823 11082 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:21:53.416070 11074 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:21:53.416342 11074 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:21:53.417492 11074 hybrid_clock.cc:648] HybridClock initialized: now 1790652113417479 us; error 38 us; skew 500 ppm
Sep 29 03:21:53 dist-test-slave-c9qh krb5kdc[10911](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652113, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:21:53.419715 11074 init.cc:377] Logged in from keytab as kudu/127.20.178.124@KRBTEST.COM (short username kudu)
I20260929 03:21:53.420142 11074 webserver.cc:460] Webserver started at http://127.20.178.124:37751/ using document root <none> and password file <none>
I20260929 03:21:53.420364 11074 fs_manager.cc:359] Metadata directory not provided
I20260929 03:21:53.420428 11074 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:21:53.420572 11074 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:21:53.421586 11074 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651852476208-21193-0/minicluster-data/master-2/data/instance:
uuid: "74c33055f0094f7395163e3f9074f9e8"
format_stamp: "Formatted at 2026-09-29 03:21:53 on dist-test-slave-c9qh"
server_key: "049f855be25d7914abab7284b835c9fe"
server_key_iv: "830d45dd14dde3904f94fff2a9509df2"
server_key_version: "encryptionkey@0"
I20260929 03:21:53.421924 11074 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651852476208-21193-0/minicluster-data/master-2/wal/instance:
uuid: "74c33055f0094f7395163e3f9074f9e8"
format_stamp: "Formatted at 2026-09-29 03:21:53 on dist-test-slave-c9qh"
server_key: "049f855be25d7914abab7284b835c9fe"
server_key_iv: "830d45dd14dde3904f94fff2a9509df2"
server_key_version: "encryptionkey@0"
I20260929 03:21:53.423223 11074 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 03:21:53.423892 11089 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:21:53.424044 11074 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 03:21:53.424139 11074 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651852476208-21193-0/minicluster-data/master-2/data,/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651852476208-21193-0/minicluster-data/master-2/wal
uuid: "74c33055f0094f7395163e3f9074f9e8"
format_stamp: "Formatted at 2026-09-29 03:21:53 on dist-test-slave-c9qh"
server_key: "049f855be25d7914abab7284b835c9fe"
server_key_iv: "830d45dd14dde3904f94fff2a9509df2"
server_key_version: "encryptionkey@0"
I20260929 03:21:53.424207 11074 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651852476208-21193-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651852476208-21193-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651852476208-21193-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:21:53.433561 11067 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.178.124" port: 46467 } attempt: 2
I20260929 03:21:53.433910 11069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:53.433803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.124:46467 (local address 127.0.0.1:43212)
0929 03:21:53.433854 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0929 03:21:53.433886 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.178.124:46467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260929 03:21:53.434234 11069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:53.434146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.124:46467 (local address 127.0.0.1:43222)
0929 03:21:53.434187 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0929 03:21:53.434216 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.178.124:46467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260929 03:21:53.434319 11067 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.178.124:46467: Network error: Client connection negotiation failed: client connection to 127.20.178.124:46467: connect: Connection refused (error 111)
I20260929 03:21:53.438997 11074 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:21:53.440773 11074 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:21:53.440912 11074 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:21:53.444576 11074 rpc_server.cc:307] RPC server started. Bound to: 127.20.178.124:46467
I20260929 03:21:53.444655 11141 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.178.124:46467 every 8 connection(s)
I20260929 03:21:53.444988 11074 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651852476208-21193-0/minicluster-data/master-2/data/info.pb
I20260929 03:21:53.445772 11142 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:21:53.447248 11142 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.178.126" port: 35917 } has no permanent_uuid. Determining permanent_uuid...
I20260929 03:21:53.451261 21193 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuoaHbX/build/release/bin/kudu as pid 11074
I20260929 03:21:53.451361 21193 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651852476208-21193-0/minicluster-data/master-2/wal/instance
I20260929 03:21:53.451546 21193 external_mini_cluster.cc:1428] Setting key 2eb5af71c877533e818158ae921fe3d4
Sep 29 03:21:53 dist-test-slave-c9qh krb5kdc[10911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652113, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.124@KRBTEST.COM for kudu/127.20.178.126@KRBTEST.COM
I20260929 03:21:53.455955 10995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:53.447886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47780 (local address 127.20.178.126:35917)
0929 03:21:53.447940 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0929 03:21:53.447943 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:21:53.448153 (+ 210us) server_negotiation.cc:407] Connection header received
0929 03:21:53.448589 (+ 436us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:21:53.448590 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:21:53.448599 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:21:53.448711 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:21:53.449271 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:53.449727 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:53.450614 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:53.450784 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:53.450834 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:21:53.453248 (+ 2414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:21:53.453249 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:21:53.453250 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:21:53.453257 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:21:53.454572 (+ 1315us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:21:53.454983 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:21:53.454985 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:21:53.454985 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:21:53.455048 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:21:53.455284 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:21:53.455285 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:21:53.455285 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:21:53.455435 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:21:53.455505 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0929 03:21:53.455831 (+ 326us) server_negotiation.cc:299] Negotiation successful
0929 03:21:53.455893 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260929 03:21:53.456205 11144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:53.447863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:35917 (local address 127.0.0.1:47780)
0929 03:21:53.448034 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0929 03:21:53.448074 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 03:21:53.448512 (+ 438us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:21:53.448785 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:21:53.448788 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:21:53.449021 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:21:53.449203 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:21:53.449207 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:21:53.449799 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:21:53.449800 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:21:53.450543 (+ 743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:21:53.450544 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:21:53.450849 (+ 305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:21:53.450850 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:21:53.450941 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:21:53.451622 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:21:53.451631 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:21:53.453151 (+ 1520us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:21:53.454670 (+ 1519us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:21:53.454677 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:21:53.454679 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:21:53.454894 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:21:53.455114 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:21:53.455115 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:21:53.455115 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:21:53.455217 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:21:53.455527 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:21:53.455533 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:21:53.455687 (+ 154us) client_negotiation.cc:769] Sending connection context
0929 03:21:53.455807 (+ 120us) client_negotiation.cc:240] Negotiation successful
0929 03:21:53.455955 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":91,"threads_started":1}
I20260929 03:21:53.456851 11142 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.178.125" port: 33321 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 03:21:53 dist-test-slave-c9qh krb5kdc[10911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652112, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.178.124@KRBTEST.COM
I20260929 03:21:53.460733 11146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:53.451910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43230 (local address 127.20.178.124:46467)
0929 03:21:53.452178 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0929 03:21:53.452182 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:21:53.452197 (+ 15us) server_negotiation.cc:407] Connection header received
0929 03:21:53.452212 (+ 15us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:21:53.452214 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:21:53.452228 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:21:53.452302 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:21:53.453003 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:53.454021 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:53.454754 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:53.455065 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:53.455141 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:21:53.457749 (+ 2608us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:21:53.457756 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:21:53.457756 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:21:53.457763 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:21:53.459513 (+ 1750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:21:53.459900 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:21:53.459903 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:21:53.459903 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:21:53.459960 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:21:53.460160 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:21:53.460161 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:21:53.460161 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:21:53.460307 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:21:53.460477 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0929 03:21:53.460522 (+ 45us) server_negotiation.cc:299] Negotiation successful
0929 03:21:53.460593 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":245,"thread_start_us":75,"threads_started":1}
Sep 29 03:21:53 dist-test-slave-c9qh krb5kdc[10911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652113, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.124@KRBTEST.COM for kudu/127.20.178.125@KRBTEST.COM
I20260929 03:21:53.463801 11144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:53.457094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.125:33321 (local address 127.0.0.1:54394)
0929 03:21:53.457142 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0929 03:21:53.457153 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 03:21:53.457195 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:21:53.457349 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:21:53.457349 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:21:53.457493 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:21:53.457561 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:21:53.457562 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:21:53.458395 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:21:53.458395 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:21:53.458860 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:21:53.458861 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:21:53.459340 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:21:53.459341 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:21:53.459389 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:21:53.459837 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:21:53.459842 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:21:53.461276 (+ 1434us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:21:53.462910 (+ 1634us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:21:53.462911 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:21:53.462911 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:21:53.463088 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:21:53.463278 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:21:53.463278 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:21:53.463278 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:21:53.463326 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:21:53.463559 (+ 233us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:21:53.463559 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:21:53.463589 (+ 30us) client_negotiation.cc:769] Sending connection context
0929 03:21:53.463638 (+ 49us) client_negotiation.cc:240] Negotiation successful
0929 03:21:53.463698 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260929 03:21:53.463827 11071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:53.457176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54394 (local address 127.20.178.125:33321)
0929 03:21:53.457242 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0929 03:21:53.457245 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:21:53.457254 (+ 9us) server_negotiation.cc:407] Connection header received
0929 03:21:53.457280 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:21:53.457281 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:21:53.457291 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:21:53.457345 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:21:53.457637 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:53.458322 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:53.458938 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:53.459266 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:53.459343 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:21:53.461360 (+ 2017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:21:53.461361 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:21:53.461361 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:21:53.461368 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:21:53.462836 (+ 1468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:21:53.463152 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:21:53.463153 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:21:53.463154 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:21:53.463213 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:21:53.463380 (+ 167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:21:53.463381 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:21:53.463381 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:21:53.463500 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:21:53.463563 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0929 03:21:53.463660 (+ 97us) server_negotiation.cc:299] Negotiation successful
0929 03:21:53.463738 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260929 03:21:53.464138 11142 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.178.124" port: 46467 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 03:21:53 dist-test-slave-c9qh krb5kdc[10911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652112, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.178.126@KRBTEST.COM
Sep 29 03:21:53 dist-test-slave-c9qh krb5kdc[10911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652113, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.124@KRBTEST.COM for kudu/127.20.178.124@KRBTEST.COM
I20260929 03:21:53.469259 10995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:53.461561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47792 (local address 127.20.178.126:35917)
0929 03:21:53.461864 (+ 303us) server_negotiation.cc:206] Beginning negotiation
0929 03:21:53.461866 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:21:53.461874 (+ 8us) server_negotiation.cc:407] Connection header received
0929 03:21:53.461894 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:21:53.461895 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:21:53.461905 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:21:53.461970 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:21:53.462412 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:53.462848 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:53.463751 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:53.464109 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:53.464168 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:21:53.466750 (+ 2582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:21:53.466752 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:21:53.466752 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:21:53.466758 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:21:53.468134 (+ 1376us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:21:53.468497 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:21:53.468499 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:21:53.468499 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:21:53.468551 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:21:53.468765 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:21:53.468766 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:21:53.468767 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:21:53.468873 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:21:53.469059 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0929 03:21:53.469095 (+ 36us) server_negotiation.cc:299] Negotiation successful
0929 03:21:53.469164 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280}
I20260929 03:21:53.471300 11144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:53.464349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.124:46467 (local address 127.0.0.1:43242)
0929 03:21:53.464430 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0929 03:21:53.464439 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 03:21:53.464479 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:21:53.464595 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:21:53.464596 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:21:53.464758 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:21:53.464784 (+ 26us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:21:53.464785 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:21:53.465695 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:21:53.465696 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:21:53.466446 (+ 750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:21:53.466448 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:21:53.466907 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:21:53.466908 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:21:53.466965 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:21:53.467254 (+ 289us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:21:53.467259 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:21:53.468583 (+ 1324us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:21:53.470321 (+ 1738us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:21:53.470322 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:21:53.470322 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:21:53.470485 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:21:53.470680 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:21:53.470681 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:21:53.470681 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:21:53.470738 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:21:53.470984 (+ 246us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:21:53.470985 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:21:53.471028 (+ 43us) client_negotiation.cc:769] Sending connection context
0929 03:21:53.471104 (+ 76us) client_negotiation.cc:240] Negotiation successful
0929 03:21:53.471194 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69}
I20260929 03:21:53.471352 11146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:53.464397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43242 (local address 127.20.178.124:46467)
0929 03:21:53.464432 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0929 03:21:53.464433 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 03:21:53.464467 (+ 34us) server_negotiation.cc:407] Connection header received
0929 03:21:53.464540 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:21:53.464541 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:21:53.464548 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:21:53.464589 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:21:53.464861 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:53.465621 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:53.466517 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:53.466824 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:53.466888 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:21:53.468671 (+ 1783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:21:53.468672 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:21:53.468672 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:21:53.468678 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:21:53.470251 (+ 1573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:21:53.470566 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:21:53.470567 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:21:53.470568 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:21:53.470615 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:21:53.470821 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:21:53.470822 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:21:53.470823 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:21:53.470923 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:21:53.470991 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0929 03:21:53.471137 (+ 146us) server_negotiation.cc:299] Negotiation successful
0929 03:21:53.471241 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":40,"server-negotiator.queue_time_us":18}
I20260929 03:21:53.473093 11142 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74c33055f0094f7395163e3f9074f9e8: Bootstrap starting.
Sep 29 03:21:53 dist-test-slave-c9qh krb5kdc[10911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652112, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.178.125@KRBTEST.COM
I20260929 03:21:53.473994 11142 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 74c33055f0094f7395163e3f9074f9e8: Neither blocks nor log segments found. Creating new log.
I20260929 03:21:53.474277 11142 log.cc:824] T 00000000000000000000000000000000 P 74c33055f0094f7395163e3f9074f9e8: Log is configured to *not* fsync() on all Append() calls
I20260929 03:21:53.474980 11142 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74c33055f0094f7395163e3f9074f9e8: No bootstrap required, opened a new log
I20260929 03:21:53.476325 11142 raft_consensus.cc:348] T 00000000000000000000000000000000 P 74c33055f0094f7395163e3f9074f9e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57f8519c269e456dbab3c7fa4665d876" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 35917 } } peers { permanent_uuid: "dda8ab636c9047e28e7d528c7987af6b" member_type: VOTER last_known_addr { host: "127.20.178.125" port: 33321 } } peers { permanent_uuid: "74c33055f0094f7395163e3f9074f9e8" member_type: VOTER last_known_addr { host: "127.20.178.124" port: 46467 } }
I20260929 03:21:53.476465 11142 raft_consensus.cc:729] T 00000000000000000000000000000000 P 74c33055f0094f7395163e3f9074f9e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74c33055f0094f7395163e3f9074f9e8, State: Initialized, Role: FOLLOWER
I20260929 03:21:53.476567 11071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:53.469864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54408 (local address 127.20.178.125:33321)
0929 03:21:53.469908 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0929 03:21:53.469910 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:21:53.469917 (+ 7us) server_negotiation.cc:407] Connection header received
0929 03:21:53.469938 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:21:53.469939 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:21:53.469948 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:21:53.470021 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:21:53.470429 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:53.470876 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:53.471603 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:53.471840 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:53.471895 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:21:53.474162 (+ 2267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:21:53.474163 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:21:53.474164 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:21:53.474171 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:21:53.475521 (+ 1350us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:21:53.475840 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:21:53.475841 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:21:53.475841 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:21:53.475895 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:21:53.476080 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:21:53.476081 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:21:53.476081 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:21:53.476180 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:21:53.476362 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0929 03:21:53.476396 (+ 34us) server_negotiation.cc:299] Negotiation successful
0929 03:21:53.476473 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260929 03:21:53.476814 11142 consensus_queue.cc:260] T 00000000000000000000000000000000 P 74c33055f0094f7395163e3f9074f9e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57f8519c269e456dbab3c7fa4665d876" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 35917 } } peers { permanent_uuid: "dda8ab636c9047e28e7d528c7987af6b" member_type: VOTER last_known_addr { host: "127.20.178.125" port: 33321 } } peers { permanent_uuid: "74c33055f0094f7395163e3f9074f9e8" member_type: VOTER last_known_addr { host: "127.20.178.124" port: 46467 } }
I20260929 03:21:53.477149 11142 sys_catalog.cc:564] T 00000000000000000000000000000000 P 74c33055f0094f7395163e3f9074f9e8 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:21:53.477135 11148 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74c33055f0094f7395163e3f9074f9e8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57f8519c269e456dbab3c7fa4665d876" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 35917 } } peers { permanent_uuid: "dda8ab636c9047e28e7d528c7987af6b" member_type: VOTER last_known_addr { host: "127.20.178.125" port: 33321 } } peers { permanent_uuid: "74c33055f0094f7395163e3f9074f9e8" member_type: VOTER last_known_addr { host: "127.20.178.124" port: 46467 } } }
I20260929 03:21:53.477263 11148 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74c33055f0094f7395163e3f9074f9e8 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 03:21:53.478096 21193 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260929 03:21:53.478981 11159 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 74c33055f0094f7395163e3f9074f9e8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 03:21:53.479087 11159 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 03:21:53.479285 10966 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "57f8519c269e456dbab3c7fa4665d876"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47792
I20260929 03:21:53.479239 11074 master_runner.cc:186] Error getting master registration for 127.20.178.126:35917: OK, instance_id { permanent_uuid: "57f8519c269e456dbab3c7fa4665d876" instance_seqno: 1790652113139647 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 03:21:53.480772 11074 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 03:21:53.481163 10992 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.178.124" port: 46467 } attempt: 2
Sep 29 03:21:53 dist-test-slave-c9qh krb5kdc[10911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652113, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.126@KRBTEST.COM for kudu/127.20.178.124@KRBTEST.COM
I20260929 03:21:53.488266 10994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:53.481370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.124:46467 (local address 127.0.0.1:43252)
0929 03:21:53.481416 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0929 03:21:53.481426 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 03:21:53.481462 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:21:53.481594 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:21:53.481595 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:21:53.481746 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:21:53.481787 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:21:53.481788 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:21:53.482633 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:21:53.482634 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:21:53.483404 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:21:53.483405 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:21:53.483878 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:21:53.483878 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:21:53.483942 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:21:53.484334 (+ 392us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:21:53.484338 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:21:53.485461 (+ 1123us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:21:53.487185 (+ 1724us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:21:53.487186 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:21:53.487187 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:21:53.487422 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:21:53.487621 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:21:53.487621 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:21:53.487622 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:21:53.487682 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:21:53.487971 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:21:53.487971 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:21:53.488019 (+ 48us) client_negotiation.cc:769] Sending connection context
0929 03:21:53.488082 (+ 63us) client_negotiation.cc:240] Negotiation successful
0929 03:21:53.488172 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260929 03:21:53.488284 11146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:53.481405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43252 (local address 127.20.178.124:46467)
0929 03:21:53.481462 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0929 03:21:53.481464 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:21:53.481472 (+ 8us) server_negotiation.cc:407] Connection header received
0929 03:21:53.481523 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:21:53.481524 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:21:53.481532 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:21:53.481592 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:21:53.481856 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:53.482545 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:53.483480 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:53.483792 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:53.483857 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:21:53.485554 (+ 1697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:21:53.485555 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:21:53.485556 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:21:53.485562 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:21:53.487090 (+ 1528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:21:53.487493 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:21:53.487494 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:21:53.487494 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:21:53.487551 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:21:53.487744 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:21:53.487745 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:21:53.487745 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:21:53.487896 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:21:53.487962 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0929 03:21:53.488118 (+ 156us) server_negotiation.cc:299] Negotiation successful
0929 03:21:53.488203 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260929 03:21:53.489583 10992 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57f8519c269e456dbab3c7fa4665d876: Bootstrap starting.
I20260929 03:21:53.490140 10992 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 57f8519c269e456dbab3c7fa4665d876: Neither blocks nor log segments found. Creating new log.
I20260929 03:21:53.490409 10992 log.cc:824] T 00000000000000000000000000000000 P 57f8519c269e456dbab3c7fa4665d876: Log is configured to *not* fsync() on all Append() calls
I20260929 03:21:53.491061 10992 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57f8519c269e456dbab3c7fa4665d876: No bootstrap required, opened a new log
I20260929 03:21:53.491986 10992 raft_consensus.cc:348] T 00000000000000000000000000000000 P 57f8519c269e456dbab3c7fa4665d876 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57f8519c269e456dbab3c7fa4665d876" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 35917 } } peers { permanent_uuid: "dda8ab636c9047e28e7d528c7987af6b" member_type: VOTER last_known_addr { host: "127.20.178.125" port: 33321 } } peers { permanent_uuid: "74c33055f0094f7395163e3f9074f9e8" member_type: VOTER last_known_addr { host: "127.20.178.124" port: 46467 } }
I20260929 03:21:53.492128 10992 raft_consensus.cc:729] T 00000000000000000000000000000000 P 57f8519c269e456dbab3c7fa4665d876 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57f8519c269e456dbab3c7fa4665d876, State: Initialized, Role: FOLLOWER
I20260929 03:21:53.492260 10992 consensus_queue.cc:260] T 00000000000000000000000000000000 P 57f8519c269e456dbab3c7fa4665d876 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57f8519c269e456dbab3c7fa4665d876" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 35917 } } peers { permanent_uuid: "dda8ab636c9047e28e7d528c7987af6b" member_type: VOTER last_known_addr { host: "127.20.178.125" port: 33321 } } peers { permanent_uuid: "74c33055f0094f7395163e3f9074f9e8" member_type: VOTER last_known_addr { host: "127.20.178.124" port: 46467 } }
I20260929 03:21:53.492530 11162 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57f8519c269e456dbab3c7fa4665d876 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57f8519c269e456dbab3c7fa4665d876" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 35917 } } peers { permanent_uuid: "dda8ab636c9047e28e7d528c7987af6b" member_type: VOTER last_known_addr { host: "127.20.178.125" port: 33321 } } peers { permanent_uuid: "74c33055f0094f7395163e3f9074f9e8" member_type: VOTER last_known_addr { host: "127.20.178.124" port: 46467 } } }
I20260929 03:21:53.492624 10966 raft_consensus.cc:482] T 00000000000000000000000000000000 P 57f8519c269e456dbab3c7fa4665d876 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260929 03:21:53.492578 10992 sys_catalog.cc:564] T 00000000000000000000000000000000 P 57f8519c269e456dbab3c7fa4665d876 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:21:53.492668 11162 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57f8519c269e456dbab3c7fa4665d876 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 03:21:53.492726 10966 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 57f8519c269e456dbab3c7fa4665d876 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:21:53.493609 10966 raft_consensus.cc:504] T 00000000000000000000000000000000 P 57f8519c269e456dbab3c7fa4665d876 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57f8519c269e456dbab3c7fa4665d876" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 35917 } } peers { permanent_uuid: "dda8ab636c9047e28e7d528c7987af6b" member_type: VOTER last_known_addr { host: "127.20.178.125" port: 33321 } } peers { permanent_uuid: "74c33055f0094f7395163e3f9074f9e8" member_type: VOTER last_known_addr { host: "127.20.178.124" port: 46467 } }
I20260929 03:21:53.494103 11116 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "57f8519c269e456dbab3c7fa4665d876" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "74c33055f0094f7395163e3f9074f9e8"
I20260929 03:21:53.494237 11116 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 74c33055f0094f7395163e3f9074f9e8 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:21:53.495066 11116 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 74c33055f0094f7395163e3f9074f9e8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57f8519c269e456dbab3c7fa4665d876 in term 1.
I20260929 03:21:53.495313 10942 leader_election.cc:302] T 00000000000000000000000000000000 P 57f8519c269e456dbab3c7fa4665d876 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57f8519c269e456dbab3c7fa4665d876, 74c33055f0094f7395163e3f9074f9e8; no voters:
I20260929 03:21:53.495491 10966 leader_election.cc:288] T 00000000000000000000000000000000 P 57f8519c269e456dbab3c7fa4665d876 [CANDIDATE]: Term 1 election: Requested vote from peers dda8ab636c9047e28e7d528c7987af6b (127.20.178.125:33321), 74c33055f0094f7395163e3f9074f9e8 (127.20.178.124:46467)
I20260929 03:21:53.495662 11162 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 57f8519c269e456dbab3c7fa4665d876 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:21:53.495713 11162 raft_consensus.cc:686] T 00000000000000000000000000000000 P 57f8519c269e456dbab3c7fa4665d876 [term 1 LEADER]: Becoming Leader. State: Replica: 57f8519c269e456dbab3c7fa4665d876, State: Running, Role: LEADER
I20260929 03:21:53.495771 11162 consensus_queue.cc:237] T 00000000000000000000000000000000 P 57f8519c269e456dbab3c7fa4665d876 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57f8519c269e456dbab3c7fa4665d876" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 35917 } } peers { permanent_uuid: "dda8ab636c9047e28e7d528c7987af6b" member_type: VOTER last_known_addr { host: "127.20.178.125" port: 33321 } } peers { permanent_uuid: "74c33055f0094f7395163e3f9074f9e8" member_type: VOTER last_known_addr { host: "127.20.178.124" port: 46467 } }
I20260929 03:21:53.496562 11162 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57f8519c269e456dbab3c7fa4665d876 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 57f8519c269e456dbab3c7fa4665d876. Latest consensus state: current_term: 1 leader_uuid: "57f8519c269e456dbab3c7fa4665d876" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57f8519c269e456dbab3c7fa4665d876" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 35917 } } peers { permanent_uuid: "dda8ab636c9047e28e7d528c7987af6b" member_type: VOTER last_known_addr { host: "127.20.178.125" port: 33321 } } peers { permanent_uuid: "74c33055f0094f7395163e3f9074f9e8" member_type: VOTER last_known_addr { host: "127.20.178.124" port: 46467 } } }
I20260929 03:21:53.496670 11162 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57f8519c269e456dbab3c7fa4665d876 [sys.catalog]: This master's current role is: LEADER
W20260929 03:21:53.497737 11173 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 57f8519c269e456dbab3c7fa4665d876: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 03:21:53.497794 11173 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 03:21:53.496829 10924 master_runner.cc:186] Error getting master registration for 127.20.178.125:33321: OK, instance_id { permanent_uuid: "dda8ab636c9047e28e7d528c7987af6b" instance_seqno: 1790652113292922 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 03:21:53.498986 10924 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 03:21:53.500756 11180 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:21:53.500974 11180 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:21:53.506305 11116 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 74c33055f0094f7395163e3f9074f9e8 [term 1 FOLLOWER]: Refusing update from remote peer 57f8519c269e456dbab3c7fa4665d876: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:21:53.506685 11174 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 57f8519c269e456dbab3c7fa4665d876 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74c33055f0094f7395163e3f9074f9e8" member_type: VOTER last_known_addr { host: "127.20.178.124" port: 46467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:21:53.507858 11148 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74c33055f0094f7395163e3f9074f9e8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 57f8519c269e456dbab3c7fa4665d876. Latest consensus state: current_term: 1 leader_uuid: "57f8519c269e456dbab3c7fa4665d876" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57f8519c269e456dbab3c7fa4665d876" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 35917 } } peers { permanent_uuid: "dda8ab636c9047e28e7d528c7987af6b" member_type: VOTER last_known_addr { host: "127.20.178.125" port: 33321 } } peers { permanent_uuid: "74c33055f0094f7395163e3f9074f9e8" member_type: VOTER last_known_addr { host: "127.20.178.124" port: 46467 } } }
I20260929 03:21:53.507937 11148 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74c33055f0094f7395163e3f9074f9e8 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 03:21:53.510957 10995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:53.497042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47804 (local address 127.20.178.126:35917)
0929 03:21:53.497075 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0929 03:21:53.497077 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:21:53.497860 (+ 783us) server_negotiation.cc:407] Connection header received
0929 03:21:53.498043 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:21:53.498044 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:21:53.498056 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:21:53.498089 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:21:53.499270 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:53.499958 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:53.501491 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:53.501781 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:53.501823 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:21:53.506520 (+ 4697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:21:53.506521 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:21:53.506521 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:21:53.506528 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:21:53.508302 (+ 1774us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:21:53.509423 (+ 1121us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:21:53.509424 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:21:53.509424 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:21:53.509496 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:21:53.510063 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:21:53.510064 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:21:53.510065 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:21:53.510204 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:21:53.510284 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0929 03:21:53.510788 (+ 504us) server_negotiation.cc:299] Negotiation successful
0929 03:21:53.510863 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260929 03:21:53.511101 11071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:53.497493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54418 (local address 127.20.178.125:33321)
0929 03:21:53.497528 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0929 03:21:53.497529 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 03:21:53.500798 (+ 3269us) server_negotiation.cc:407] Connection header received
0929 03:21:53.500960 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:21:53.500961 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:21:53.500971 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:21:53.501002 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:21:53.501809 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:53.502522 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:53.503521 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:53.504232 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:53.504274 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:21:53.508238 (+ 3964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:21:53.508239 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:21:53.508239 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:21:53.508246 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:21:53.509898 (+ 1652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:21:53.510291 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:21:53.510292 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:21:53.510292 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:21:53.510357 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:21:53.510551 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:21:53.510552 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:21:53.510552 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:21:53.510667 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:21:53.510728 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0929 03:21:53.510917 (+ 189us) server_negotiation.cc:299] Negotiation successful
0929 03:21:53.510992 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260929 03:21:53.513892 11174 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57f8519c269e456dbab3c7fa4665d876 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "57f8519c269e456dbab3c7fa4665d876" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57f8519c269e456dbab3c7fa4665d876" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 35917 } } peers { permanent_uuid: "dda8ab636c9047e28e7d528c7987af6b" member_type: VOTER last_known_addr { host: "127.20.178.125" port: 33321 } } peers { permanent_uuid: "74c33055f0094f7395163e3f9074f9e8" member_type: VOTER last_known_addr { host: "127.20.178.124" port: 46467 } } }
I20260929 03:21:53.513972 11174 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57f8519c269e456dbab3c7fa4665d876 [sys.catalog]: This master's current role is: LEADER
I20260929 03:21:53.514468 11148 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74c33055f0094f7395163e3f9074f9e8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "57f8519c269e456dbab3c7fa4665d876" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57f8519c269e456dbab3c7fa4665d876" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 35917 } } peers { permanent_uuid: "dda8ab636c9047e28e7d528c7987af6b" member_type: VOTER last_known_addr { host: "127.20.178.125" port: 33321 } } peers { permanent_uuid: "74c33055f0094f7395163e3f9074f9e8" member_type: VOTER last_known_addr { host: "127.20.178.124" port: 46467 } } }
I20260929 03:21:53.514545 11148 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74c33055f0094f7395163e3f9074f9e8 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 03:21:53.514777 11185 mvcc.cc:205] Tried to move back new op lower bound from 7334511056919506944 to 7334511056879157248. Current Snapshot: MvccSnapshot[applied={T|T < 7334511056919506944}]
I20260929 03:21:53.514904 11186 mvcc.cc:205] Tried to move back new op lower bound from 7334511056919506944 to 7334511056879157248. Current Snapshot: MvccSnapshot[applied={T|T < 7334511056919506944}]
I20260929 03:21:53.518549 11146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:53.498029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43266 (local address 127.20.178.124:46467)
0929 03:21:53.498064 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0929 03:21:53.498065 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 03:21:53.503177 (+ 5112us) server_negotiation.cc:407] Connection header received
0929 03:21:53.503456 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:21:53.503457 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:21:53.503478 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:21:53.503508 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:21:53.505245 (+ 1737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:53.505958 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:53.512117 (+ 6159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:53.512416 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:53.512456 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:21:53.515073 (+ 2617us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:21:53.515074 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:21:53.515074 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:21:53.515081 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:21:53.516855 (+ 1774us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:21:53.517487 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:21:53.517488 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:21:53.517489 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:21:53.517546 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:21:53.517854 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:21:53.517855 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:21:53.517855 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:21:53.517986 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:21:53.518070 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0929 03:21:53.518392 (+ 322us) server_negotiation.cc:299] Negotiation successful
0929 03:21:53.518464 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260929 03:21:53.518925 11180 catalog_manager.cc:1347] Generated new cluster ID: c01e6c99feed41bfb4fcf44bbd6da1e9
I20260929 03:21:53.518996 11180 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:21:53.525447 11067 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.178.124" port: 46467 } attempt: 3
Sep 29 03:21:53 dist-test-slave-c9qh krb5kdc[10911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652113, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.125@KRBTEST.COM for kudu/127.20.178.124@KRBTEST.COM
I20260929 03:21:53.534909 11069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:53.525682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.124:46467 (local address 127.0.0.1:43274)
0929 03:21:53.525760 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0929 03:21:53.525769 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 03:21:53.525805 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:21:53.526077 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:21:53.526078 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:21:53.526240 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:21:53.526300 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:21:53.526302 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:21:53.527240 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:21:53.527241 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:21:53.528063 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:21:53.528066 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:21:53.528692 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:21:53.528693 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:21:53.528763 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:21:53.529300 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:21:53.529305 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:21:53.531510 (+ 2205us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:21:53.533552 (+ 2042us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:21:53.533553 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:21:53.533554 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:21:53.533772 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:21:53.534121 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:21:53.534122 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:21:53.534122 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:21:53.534190 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:21:53.534604 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:21:53.534605 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:21:53.534659 (+ 54us) client_negotiation.cc:769] Sending connection context
0929 03:21:53.534705 (+ 46us) client_negotiation.cc:240] Negotiation successful
0929 03:21:53.534796 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260929 03:21:53.534916 11180 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:21:53.535609 11180 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:21:53.536417 11146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:53.525884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43274 (local address 127.20.178.124:46467)
0929 03:21:53.525939 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0929 03:21:53.525941 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:21:53.525950 (+ 9us) server_negotiation.cc:407] Connection header received
0929 03:21:53.525969 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:21:53.525970 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:21:53.525979 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:21:53.526035 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:21:53.526412 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:53.527153 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:53.528185 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:53.528530 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:53.528598 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:21:53.531615 (+ 3017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:21:53.531616 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:21:53.531617 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:21:53.531623 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:21:53.533447 (+ 1824us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:21:53.533873 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:21:53.533874 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:21:53.533874 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:21:53.533992 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:21:53.534260 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:21:53.534261 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:21:53.534261 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:21:53.534517 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:21:53.536069 (+ 1552us) server_negotiation.cc:1035] Waiting for connection context
0929 03:21:53.536139 (+ 70us) server_negotiation.cc:299] Negotiation successful
0929 03:21:53.536274 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260929 03:21:53.538221 11067 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dda8ab636c9047e28e7d528c7987af6b: Bootstrap starting.
I20260929 03:21:53.538924 11067 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dda8ab636c9047e28e7d528c7987af6b: Neither blocks nor log segments found. Creating new log.
I20260929 03:21:53.539233 11067 log.cc:824] T 00000000000000000000000000000000 P dda8ab636c9047e28e7d528c7987af6b: Log is configured to *not* fsync() on all Append() calls
I20260929 03:21:53.540248 11067 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dda8ab636c9047e28e7d528c7987af6b: No bootstrap required, opened a new log
I20260929 03:21:53.542263 11067 raft_consensus.cc:348] T 00000000000000000000000000000000 P dda8ab636c9047e28e7d528c7987af6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57f8519c269e456dbab3c7fa4665d876" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 35917 } } peers { permanent_uuid: "dda8ab636c9047e28e7d528c7987af6b" member_type: VOTER last_known_addr { host: "127.20.178.125" port: 33321 } } peers { permanent_uuid: "74c33055f0094f7395163e3f9074f9e8" member_type: VOTER last_known_addr { host: "127.20.178.124" port: 46467 } }
I20260929 03:21:53.542390 11067 raft_consensus.cc:729] T 00000000000000000000000000000000 P dda8ab636c9047e28e7d528c7987af6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dda8ab636c9047e28e7d528c7987af6b, State: Initialized, Role: FOLLOWER
I20260929 03:21:53.542465 11067 consensus_queue.cc:260] T 00000000000000000000000000000000 P dda8ab636c9047e28e7d528c7987af6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57f8519c269e456dbab3c7fa4665d876" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 35917 } } peers { permanent_uuid: "dda8ab636c9047e28e7d528c7987af6b" member_type: VOTER last_known_addr { host: "127.20.178.125" port: 33321 } } peers { permanent_uuid: "74c33055f0094f7395163e3f9074f9e8" member_type: VOTER last_known_addr { host: "127.20.178.124" port: 46467 } }
I20260929 03:21:53.542778 11067 sys_catalog.cc:564] T 00000000000000000000000000000000 P dda8ab636c9047e28e7d528c7987af6b [sys.catalog]: configured and running, proceeding with master startup.
W20260929 03:21:53.544345 11020 tablet.cc:2369] T 00000000000000000000000000000000 P dda8ab636c9047e28e7d528c7987af6b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 03:21:53.544086 11040 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dda8ab636c9047e28e7d528c7987af6b [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:21:53.545529 11040 raft_consensus.cc:1261] T 00000000000000000000000000000000 P dda8ab636c9047e28e7d528c7987af6b [term 1 FOLLOWER]: Refusing update from remote peer 57f8519c269e456dbab3c7fa4665d876: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:21:53.545570 11041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "57f8519c269e456dbab3c7fa4665d876" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "dda8ab636c9047e28e7d528c7987af6b"
I20260929 03:21:53.545820 11174 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 57f8519c269e456dbab3c7fa4665d876 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dda8ab636c9047e28e7d528c7987af6b" member_type: VOTER last_known_addr { host: "127.20.178.125" port: 33321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:21:53.546840 11041 raft_consensus.cc:2454] T 00000000000000000000000000000000 P dda8ab636c9047e28e7d528c7987af6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57f8519c269e456dbab3c7fa4665d876 in term 1.
I20260929 03:21:53.548418 11180 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 57f8519c269e456dbab3c7fa4665d876: Generated new TSK 0
I20260929 03:21:53.548590 11180 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 03:21:53.548547 11191 sys_catalog.cc:455] T 00000000000000000000000000000000 P dda8ab636c9047e28e7d528c7987af6b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "57f8519c269e456dbab3c7fa4665d876" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57f8519c269e456dbab3c7fa4665d876" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 35917 } } peers { permanent_uuid: "dda8ab636c9047e28e7d528c7987af6b" member_type: VOTER last_known_addr { host: "127.20.178.125" port: 33321 } } peers { permanent_uuid: "74c33055f0094f7395163e3f9074f9e8" member_type: VOTER last_known_addr { host: "127.20.178.124" port: 46467 } } }
I20260929 03:21:53.548759 11191 sys_catalog.cc:458] T 00000000000000000000000000000000 P dda8ab636c9047e28e7d528c7987af6b [sys.catalog]: This master's current role is: FOLLOWER
I20260929 03:21:53.548833 11191 sys_catalog.cc:455] T 00000000000000000000000000000000 P dda8ab636c9047e28e7d528c7987af6b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 57f8519c269e456dbab3c7fa4665d876. Latest consensus state: current_term: 1 leader_uuid: "57f8519c269e456dbab3c7fa4665d876" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57f8519c269e456dbab3c7fa4665d876" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 35917 } } peers { permanent_uuid: "dda8ab636c9047e28e7d528c7987af6b" member_type: VOTER last_known_addr { host: "127.20.178.125" port: 33321 } } peers { permanent_uuid: "74c33055f0094f7395163e3f9074f9e8" member_type: VOTER last_known_addr { host: "127.20.178.124" port: 46467 } } }
I20260929 03:21:53.548902 11191 sys_catalog.cc:458] T 00000000000000000000000000000000 P dda8ab636c9047e28e7d528c7987af6b [sys.catalog]: This master's current role is: FOLLOWER
I20260929 03:21:53.548940 11191 sys_catalog.cc:455] T 00000000000000000000000000000000 P dda8ab636c9047e28e7d528c7987af6b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "57f8519c269e456dbab3c7fa4665d876" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57f8519c269e456dbab3c7fa4665d876" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 35917 } } peers { permanent_uuid: "dda8ab636c9047e28e7d528c7987af6b" member_type: VOTER last_known_addr { host: "127.20.178.125" port: 33321 } } peers { permanent_uuid: "74c33055f0094f7395163e3f9074f9e8" member_type: VOTER last_known_addr { host: "127.20.178.124" port: 46467 } } }
I20260929 03:21:53.548995 11191 sys_catalog.cc:458] T 00000000000000000000000000000000 P dda8ab636c9047e28e7d528c7987af6b [sys.catalog]: This master's current role is: FOLLOWER
W20260929 03:21:53.553448 11202 catalog_manager.cc:1558] T 00000000000000000000000000000000 P dda8ab636c9047e28e7d528c7987af6b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 03:21:53.553504 11202 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 03:21:53.554358 11204 mvcc.cc:205] Tried to move back new op lower bound from 7334511057033330688 to 7334511056879157248. Current Snapshot: MvccSnapshot[applied={T|T < 7334511056919506944}]
I20260929 03:21:53.554857 11162 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57f8519c269e456dbab3c7fa4665d876 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "57f8519c269e456dbab3c7fa4665d876" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57f8519c269e456dbab3c7fa4665d876" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 35917 } } peers { permanent_uuid: "dda8ab636c9047e28e7d528c7987af6b" member_type: VOTER last_known_addr { host: "127.20.178.125" port: 33321 } } peers { permanent_uuid: "74c33055f0094f7395163e3f9074f9e8" member_type: VOTER last_known_addr { host: "127.20.178.124" port: 46467 } } }
I20260929 03:21:53.554939 11162 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57f8519c269e456dbab3c7fa4665d876 [sys.catalog]: This master's current role is: LEADER
W20260929 03:21:53.571718 11146 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:21:53.570083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43290 (local address 127.20.178.124:46467)
0929 03:21:53.570126 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0929 03:21:53.570129 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:21:53.571071 (+ 942us) server_negotiation.cc:407] Connection header received
0929 03:21:53.571173 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:21:53.571174 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:21:53.571185 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:21:53.571216 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:21:53.571656 (+ 440us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43290: BlockingRecv error: recv got EOF from 127.0.0.1:43290 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260929 03:21:53.572587 10995 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:21:53.570517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47816 (local address 127.20.178.126:35917)
0929 03:21:53.570557 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0929 03:21:53.570558 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 03:21:53.572055 (+ 1497us) server_negotiation.cc:407] Connection header received
0929 03:21:53.572148 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:21:53.572149 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:21:53.572158 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:21:53.572187 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:21:53.572516 (+ 329us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47816: BlockingRecv error: recv got EOF from 127.0.0.1:47816 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260929 03:21:53.573469 11071 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:21:53.570825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54432 (local address 127.20.178.125:33321)
0929 03:21:53.570868 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0929 03:21:53.570870 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:21:53.572932 (+ 2062us) server_negotiation.cc:407] Connection header received
0929 03:21:53.573030 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:21:53.573031 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:21:53.573053 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:21:53.573083 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:21:53.573404 (+ 321us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54432: BlockingRecv error: recv got EOF from 127.0.0.1:54432 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260929 03:21:53.575780 11071 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:21:53.575060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54434 (local address 127.20.178.125:33321)
0929 03:21:53.575092 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0929 03:21:53.575094 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:21:53.575262 (+ 168us) server_negotiation.cc:407] Connection header received
0929 03:21:53.575365 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:21:53.575366 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:21:53.575383 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:21:53.575414 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:21:53.575737 (+ 323us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54434: BlockingRecv error: recv got EOF from 127.0.0.1:54434 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260929 03:21:53.578685 11071 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:21:53.578032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54448 (local address 127.20.178.125:33321)
0929 03:21:53.578071 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0929 03:21:53.578072 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 03:21:53.578250 (+ 178us) server_negotiation.cc:407] Connection header received
0929 03:21:53.578331 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:21:53.578332 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:21:53.578340 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:21:53.578365 (+ 25us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:21:53.578648 (+ 283us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54448: BlockingRecv error: recv got EOF from 127.0.0.1:54448 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260929 03:21:53.582436 11071 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:21:53.581772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54450 (local address 127.20.178.125:33321)
0929 03:21:53.581802 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0929 03:21:53.581803 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 03:21:53.581998 (+ 195us) server_negotiation.cc:407] Connection header received
0929 03:21:53.582084 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:21:53.582085 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:21:53.582094 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:21:53.582121 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:21:53.582401 (+ 280us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54450: BlockingRecv error: recv got EOF from 127.0.0.1:54450 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
W20260929 03:21:53.588304 11071 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:21:53.587586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54452 (local address 127.20.178.125:33321)
0929 03:21:53.587630 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0929 03:21:53.587631 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 03:21:53.587831 (+ 200us) server_negotiation.cc:407] Connection header received
0929 03:21:53.587924 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:21:53.587925 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:21:53.587934 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:21:53.587964 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:21:53.588265 (+ 301us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54452: BlockingRecv error: recv got EOF from 127.0.0.1:54452 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260929 03:21:53.598456 11071 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:21:53.597639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54456 (local address 127.20.178.125:33321)
0929 03:21:53.597675 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0929 03:21:53.597677 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:21:53.597907 (+ 230us) server_negotiation.cc:407] Connection header received
0929 03:21:53.598007 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:21:53.598008 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:21:53.598019 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:21:53.598049 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:21:53.598395 (+ 346us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54456: BlockingRecv error: recv got EOF from 127.0.0.1:54456 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260929 03:21:53.616892 11071 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:21:53.615908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54470 (local address 127.20.178.125:33321)
0929 03:21:53.615954 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0929 03:21:53.615956 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:21:53.616233 (+ 277us) server_negotiation.cc:407] Connection header received
0929 03:21:53.616345 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:21:53.616346 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:21:53.616359 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:21:53.616390 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:21:53.616837 (+ 447us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54470: BlockingRecv error: recv got EOF from 127.0.0.1:54470 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260929 03:21:53.651013 11071 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:21:53.650217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54484 (local address 127.20.178.125:33321)
0929 03:21:53.650262 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0929 03:21:53.650265 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:21:53.650403 (+ 138us) server_negotiation.cc:407] Connection header received
0929 03:21:53.650510 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:21:53.650511 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:21:53.650522 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:21:53.650593 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:21:53.650927 (+ 334us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54484: BlockingRecv error: recv got EOF from 127.0.0.1:54484 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260929 03:21:53.716993 11071 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:21:53.716156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54490 (local address 127.20.178.125:33321)
0929 03:21:53.716229 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0929 03:21:53.716231 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:21:53.716354 (+ 123us) server_negotiation.cc:407] Connection header received
0929 03:21:53.716442 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:21:53.716443 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:21:53.716455 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:21:53.716527 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:21:53.716927 (+ 400us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54490: BlockingRecv error: recv got EOF from 127.0.0.1:54490 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260929 03:21:53.846866 11071 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:21:53.846108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54506 (local address 127.20.178.125:33321)
0929 03:21:53.846166 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0929 03:21:53.846168 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:21:53.846279 (+ 111us) server_negotiation.cc:407] Connection header received
0929 03:21:53.846358 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:21:53.846359 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:21:53.846372 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:21:53.846445 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:21:53.846786 (+ 341us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54506: BlockingRecv error: recv got EOF from 127.0.0.1:54506 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
W20260929 03:21:54.104748 11071 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:21:54.104091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54508 (local address 127.20.178.125:33321)
0929 03:21:54.104164 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0929 03:21:54.104166 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:21:54.104177 (+ 11us) server_negotiation.cc:407] Connection header received
0929 03:21:54.104251 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:21:54.104252 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:21:54.104262 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:21:54.104323 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:21:54.104673 (+ 350us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54508: BlockingRecv error: recv got EOF from 127.0.0.1:54508 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 03:21:54.479570 11159 catalog_manager.cc:1259] Loaded cluster ID: c01e6c99feed41bfb4fcf44bbd6da1e9
I20260929 03:21:54.479666 11159 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 74c33055f0094f7395163e3f9074f9e8: loading cluster ID for follower catalog manager: success
I20260929 03:21:54.480463 11159 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 74c33055f0094f7395163e3f9074f9e8: acquiring CA information for follower catalog manager: success
I20260929 03:21:54.480777 11159 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 74c33055f0094f7395163e3f9074f9e8: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 03:21:54.553889 11202 catalog_manager.cc:1259] Loaded cluster ID: c01e6c99feed41bfb4fcf44bbd6da1e9
I20260929 03:21:54.553978 11202 catalog_manager.cc:1552] T 00000000000000000000000000000000 P dda8ab636c9047e28e7d528c7987af6b: loading cluster ID for follower catalog manager: success
I20260929 03:21:54.554759 11202 catalog_manager.cc:1574] T 00000000000000000000000000000000 P dda8ab636c9047e28e7d528c7987af6b: acquiring CA information for follower catalog manager: success
I20260929 03:21:54.555047 11202 catalog_manager.cc:1602] T 00000000000000000000000000000000 P dda8ab636c9047e28e7d528c7987af6b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 03:21:54.620260 11275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:54.617819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54520 (local address 127.20.178.125:33321)
0929 03:21:54.617919 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0929 03:21:54.617927 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 03:21:54.618014 (+ 87us) server_negotiation.cc:407] Connection header received
0929 03:21:54.618101 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:21:54.618102 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:21:54.618151 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:21:54.618217 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:21:54.618360 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:54.618843 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:54.619548 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:54.619733 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:54.619784 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:21:54.619917 (+ 133us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:21:54.620030 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:21:54.620088 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0929 03:21:54.620129 (+ 41us) server_negotiation.cc:299] Negotiation successful
0929 03:21:54.620145 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83,"thread_start_us":55,"threads_started":1}
I20260929 03:21:54.624565 11281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:54.622157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43300 (local address 127.20.178.124:46467)
0929 03:21:54.622281 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0929 03:21:54.622284 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:21:54.622292 (+ 8us) server_negotiation.cc:407] Connection header received
0929 03:21:54.622334 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:21:54.622334 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:21:54.622382 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:21:54.622459 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:21:54.622567 (+ 108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:54.623054 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:54.623775 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:54.623993 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:54.624073 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:21:54.624231 (+ 158us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:21:54.624335 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:21:54.624410 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0929 03:21:54.624455 (+ 45us) server_negotiation.cc:299] Negotiation successful
0929 03:21:54.624475 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":53,"threads_started":1}
I20260929 03:21:54.629171 11275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:54.626376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54530 (local address 127.20.178.125:33321)
0929 03:21:54.626413 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0929 03:21:54.626414 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 03:21:54.626422 (+ 8us) server_negotiation.cc:407] Connection header received
0929 03:21:54.626467 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:21:54.626468 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:21:54.626478 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:21:54.626531 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:21:54.626718 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:54.627267 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:54.628379 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:54.628690 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:54.628747 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:21:54.628895 (+ 148us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:21:54.628925 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:21:54.628976 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0929 03:21:54.629058 (+ 82us) server_negotiation.cc:299] Negotiation successful
0929 03:21:54.629081 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260929 03:21:54.629457 11288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:54.626187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47828 (local address 127.20.178.126:35917)
0929 03:21:54.626360 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0929 03:21:54.626362 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:21:54.626371 (+ 9us) server_negotiation.cc:407] Connection header received
0929 03:21:54.626416 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:21:54.626416 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:21:54.626462 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:21:54.626541 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:21:54.626657 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:54.627441 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:54.628509 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:54.628829 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:54.628905 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:21:54.629128 (+ 223us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:21:54.629237 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:21:54.629299 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0929 03:21:54.629349 (+ 50us) server_negotiation.cc:299] Negotiation successful
0929 03:21:54.629372 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":62,"threads_started":1}
I20260929 03:21:54.629683 11281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:54.626295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43308 (local address 127.20.178.124:46467)
0929 03:21:54.626344 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0929 03:21:54.626346 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:21:54.626354 (+ 8us) server_negotiation.cc:407] Connection header received
0929 03:21:54.626376 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:21:54.626377 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:21:54.626386 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:21:54.626473 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:21:54.626606 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:54.627365 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:54.628389 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:54.628704 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:54.628898 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:21:54.628922 (+ 24us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:21:54.628951 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:21:54.629021 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0929 03:21:54.629144 (+ 123us) server_negotiation.cc:299] Negotiation successful
0929 03:21:54.629163 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260929 03:21:54.636934 21193 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuoaHbX/build/release/bin/kudu with pid 10924
I20260929 03:21:54.641867 21193 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuoaHbX/build/release/bin/kudu with pid 10999
I20260929 03:21:54.646701 21193 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuoaHbX/build/release/bin/kudu with pid 11074
2026-09-29T03:21:54Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2755 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651852476208-21193-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:21:54 dist-test-slave-c9qh krb5kdc[11292](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:21:54 dist-test-slave-c9qh krb5kdc[11292](info): set up 2 sockets
Sep 29 03:21:54 dist-test-slave-c9qh krb5kdc[11292](info): commencing operation
krb5kdc: starting...
W20260929 03:21:55.684967 21193 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651852476208-21193-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651852476208-21193-0/krb5kdc/test-user.keytab.
Sep 29 03:21:55 dist-test-slave-c9qh krb5kdc[11292](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652115, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:21:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:21:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.178.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.178.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.178.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.126.keytab.
Entry for principal kudu/127.20.178.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.126.keytab.
I20260929 03:21:55.799609 21193 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuoaHbX/build/release/bin/kudu
/tmp/dist-test-taskuoaHbX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651852476208-21193-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651852476208-21193-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651852476208-21193-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651852476208-21193-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.178.126:39889
--webserver_interface=127.20.178.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.126.keytab
--principal=kudu/127.20.178.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.178.84:35159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.178.126:39889,127.20.178.125:42593,127.20.178.124:37121
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651852476208-21193-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651852476208-21193-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651852476208-21193-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:21:55.878525 11305 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 03:21:55.878726 11305 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:21:55.878742 11305 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:21:55.878762 11305 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:21:55.879835 11305 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:21:55.879868 11305 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:21:55.879881 11305 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:21:55.879894 11305 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:21:55.879905 11305 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:21:55.881273 11305 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.178.84:35159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651852476208-21193-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651852476208-21193-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.178.126:39889,127.20.178.125:42593,127.20.178.124:37121
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.126.keytab
--principal=kudu/127.20.178.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.178.126:39889
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651852476208-21193-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.178.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11305
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651852476208-21193-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:21:55.881453 11305 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:21:55.881619 11305 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:21:55.886209 11313 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:21:55.886245 11310 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:21:55.886219 11311 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:21:55.886507 11305 server_base.cc:1034] running on GCE node
I20260929 03:21:55.886754 11305 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:21:55.887024 11305 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:21:55.888177 11305 hybrid_clock.cc:648] HybridClock initialized: now 1790652115888170 us; error 35 us; skew 500 ppm
Sep 29 03:21:55 dist-test-slave-c9qh krb5kdc[11292](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652115, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:21:55.890538 11305 init.cc:377] Logged in from keytab as kudu/127.20.178.126@KRBTEST.COM (short username kudu)
I20260929 03:21:55.890966 11305 webserver.cc:460] Webserver started at http://127.20.178.126:37289/ using document root <none> and password file <none>
I20260929 03:21:55.891185 11305 fs_manager.cc:359] Metadata directory not provided
I20260929 03:21:55.891249 11305 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:21:55.891388 11305 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:21:55.892418 11305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651852476208-21193-0/minicluster-data/master-0/data/instance:
uuid: "204339ee7e414eed9135250b17101d3b"
format_stamp: "Formatted at 2026-09-29 03:21:55 on dist-test-slave-c9qh"
server_key: "441e5178faa744654130bb5d176d60fc"
server_key_iv: "dbe5159513df755460661946c8cd70b3"
server_key_version: "encryptionkey@0"
I20260929 03:21:55.892807 11305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651852476208-21193-0/minicluster-data/master-0/wal/instance:
uuid: "204339ee7e414eed9135250b17101d3b"
format_stamp: "Formatted at 2026-09-29 03:21:55 on dist-test-slave-c9qh"
server_key: "441e5178faa744654130bb5d176d60fc"
server_key_iv: "dbe5159513df755460661946c8cd70b3"
server_key_version: "encryptionkey@0"
I20260929 03:21:55.894126 11305 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260929 03:21:55.894824 11320 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:21:55.895030 11305 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 03:21:55.895141 11305 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651852476208-21193-0/minicluster-data/master-0/data,/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651852476208-21193-0/minicluster-data/master-0/wal
uuid: "204339ee7e414eed9135250b17101d3b"
format_stamp: "Formatted at 2026-09-29 03:21:55 on dist-test-slave-c9qh"
server_key: "441e5178faa744654130bb5d176d60fc"
server_key_iv: "dbe5159513df755460661946c8cd70b3"
server_key_version: "encryptionkey@0"
I20260929 03:21:55.895221 11305 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651852476208-21193-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651852476208-21193-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651852476208-21193-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:21:55.913762 11305 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:21:55.915666 11305 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:21:55.915835 11305 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:21:55.919659 11305 rpc_server.cc:307] RPC server started. Bound to: 127.20.178.126:39889
I20260929 03:21:55.919708 11372 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.178.126:39889 every 8 connection(s)
I20260929 03:21:55.920069 11305 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651852476208-21193-0/minicluster-data/master-0/data/info.pb
I20260929 03:21:55.920970 11373 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:21:55.922411 11373 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.178.126" port: 39889 } has no permanent_uuid. Determining permanent_uuid...
I20260929 03:21:55.924206 21193 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuoaHbX/build/release/bin/kudu as pid 11305
I20260929 03:21:55.924283 21193 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651852476208-21193-0/minicluster-data/master-0/wal/instance
I20260929 03:21:55.924463 21193 external_mini_cluster.cc:1428] Setting key 6e347b52d08d6e4f6b1a91773d474ad6
Sep 29 03:21:55 dist-test-slave-c9qh krb5kdc[11292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652115, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.126@KRBTEST.COM for kudu/127.20.178.126@KRBTEST.COM
I20260929 03:21:55.932260 11375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:55.923020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:39889 (local address 127.0.0.1:48644)
0929 03:21:55.923180 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0929 03:21:55.923215 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0929 03:21:55.923701 (+ 486us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:21:55.923950 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:21:55.923952 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:21:55.924204 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:21:55.924468 (+ 264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:21:55.924474 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:21:55.925530 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:21:55.925531 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:21:55.926187 (+ 656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:21:55.926189 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:21:55.926694 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:21:55.926695 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:21:55.926758 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:21:55.927229 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:21:55.927239 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:21:55.928856 (+ 1617us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:21:55.930884 (+ 2028us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:21:55.930891 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:21:55.930892 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:21:55.931088 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:21:55.931354 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:21:55.931354 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:21:55.931355 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:21:55.931427 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:21:55.931856 (+ 429us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:21:55.931861 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:21:55.931911 (+ 50us) client_negotiation.cc:769] Sending connection context
0929 03:21:55.931979 (+ 68us) client_negotiation.cc:240] Negotiation successful
0929 03:21:55.932104 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":79,"threads_started":1}
I20260929 03:21:55.932356 11376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:55.923046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48644 (local address 127.20.178.126:39889)
0929 03:21:55.923246 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0929 03:21:55.923249 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:21:55.923273 (+ 24us) server_negotiation.cc:407] Connection header received
0929 03:21:55.923835 (+ 562us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:21:55.923837 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:21:55.923852 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:21:55.923925 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:21:55.924588 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:55.925426 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:55.926295 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:55.926600 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:55.926721 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:21:55.928936 (+ 2215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:21:55.928944 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:21:55.928945 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:21:55.928953 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:21:55.930806 (+ 1853us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:21:55.931167 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:21:55.931171 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:21:55.931172 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:21:55.931278 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:21:55.931485 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:21:55.931486 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:21:55.931487 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:21:55.931776 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:21:55.931876 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0929 03:21:55.932037 (+ 161us) server_negotiation.cc:299] Negotiation successful
0929 03:21:55.932185 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":41,"server-negotiator.queue_time_us":175,"thread_start_us":68,"threads_started":1}
WARNING: no policy specified for kudu/127.20.178.125@KRBTEST.COM; defaulting to no policy
I20260929 03:21:55.933218 11373 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.178.125" port: 42593 } has no permanent_uuid. Determining permanent_uuid...
I20260929 03:21:55.933563 11375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:55.933454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.125:42593 (local address 127.0.0.1:57546)
0929 03:21:55.933504 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0929 03:21:55.933545 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.178.125:42593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260929 03:21:55.933688 11323 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.178.125:42593: connect: Connection refused (error 111)
I20260929 03:21:55.934075 11375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:55.933980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.125:42593 (local address 127.0.0.1:57556)
0929 03:21:55.934030 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0929 03:21:55.934054 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.178.125:42593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260929 03:21:55.934214 11373 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.178.125:42593: Network error: Client connection negotiation failed: client connection to 127.20.178.125:42593: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.178.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.178.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.125.keytab.
Entry for principal kudu/127.20.178.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.125.keytab.
I20260929 03:21:55.951260 21193 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuoaHbX/build/release/bin/kudu
/tmp/dist-test-taskuoaHbX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651852476208-21193-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651852476208-21193-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651852476208-21193-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651852476208-21193-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.178.125:42593
--webserver_interface=127.20.178.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.125.keytab
--principal=kudu/127.20.178.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.178.84:35159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.178.126:39889,127.20.178.125:42593,127.20.178.124:37121
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651852476208-21193-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651852476208-21193-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651852476208-21193-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 03:21:55.983445 11373 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.178.125" port: 42593 } attempt: 1
I20260929 03:21:55.983918 11375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:55.983791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.125:42593 (local address 127.0.0.1:57572)
0929 03:21:55.983851 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0929 03:21:55.983889 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.178.125:42593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260929 03:21:55.984247 11375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:55.984155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.125:42593 (local address 127.0.0.1:57576)
0929 03:21:55.984197 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0929 03:21:55.984228 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.178.125:42593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260929 03:21:55.984397 11373 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.178.125:42593: Network error: Client connection negotiation failed: client connection to 127.20.178.125:42593: connect: Connection refused (error 111)
W20260929 03:21:56.032164 11380 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 03:21:56.032377 11380 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:21:56.032393 11380 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:21:56.032438 11380 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:21:56.033641 11380 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:21:56.033681 11380 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:21:56.033695 11380 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:21:56.033746 11380 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:21:56.033764 11380 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:21:56.035306 11380 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.178.84:35159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651852476208-21193-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651852476208-21193-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.20.178.126:39889,127.20.178.125:42593,127.20.178.124:37121
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.125.keytab
--principal=kudu/127.20.178.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.178.125:42593
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651852476208-21193-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.20.178.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11380
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651852476208-21193-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:21:56.035539 11380 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:21:56.035744 11380 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:21:56.040475 11385 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:21:56.040514 11386 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:21:56.040485 11388 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:21:56.040688 11380 server_base.cc:1034] running on GCE node
I20260929 03:21:56.040926 11380 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:21:56.041173 11380 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:21:56.042305 11380 hybrid_clock.cc:648] HybridClock initialized: now 1790652116042280 us; error 42 us; skew 500 ppm
Sep 29 03:21:56 dist-test-slave-c9qh krb5kdc[11292](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652116, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:21:56.044615 11380 init.cc:377] Logged in from keytab as kudu/127.20.178.125@KRBTEST.COM (short username kudu)
I20260929 03:21:56.045113 11380 webserver.cc:460] Webserver started at http://127.20.178.125:33157/ using document root <none> and password file <none>
I20260929 03:21:56.045301 11380 fs_manager.cc:359] Metadata directory not provided
I20260929 03:21:56.045341 11380 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:21:56.045426 11380 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:21:56.046397 11380 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651852476208-21193-0/minicluster-data/master-1/data/instance:
uuid: "b2f4204b0ccd4581a2b7838e1458d704"
format_stamp: "Formatted at 2026-09-29 03:21:56 on dist-test-slave-c9qh"
server_key: "b9bc819ad3201164f33672928843cc0c"
server_key_iv: "3f38b4de95c9b30b3b3616fecc624e0e"
server_key_version: "encryptionkey@0"
I20260929 03:21:56.046687 11380 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651852476208-21193-0/minicluster-data/master-1/wal/instance:
uuid: "b2f4204b0ccd4581a2b7838e1458d704"
format_stamp: "Formatted at 2026-09-29 03:21:56 on dist-test-slave-c9qh"
server_key: "b9bc819ad3201164f33672928843cc0c"
server_key_iv: "3f38b4de95c9b30b3b3616fecc624e0e"
server_key_version: "encryptionkey@0"
I20260929 03:21:56.047858 11380 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260929 03:21:56.048496 11395 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:21:56.048673 11380 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 03:21:56.048755 11380 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651852476208-21193-0/minicluster-data/master-1/data,/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651852476208-21193-0/minicluster-data/master-1/wal
uuid: "b2f4204b0ccd4581a2b7838e1458d704"
format_stamp: "Formatted at 2026-09-29 03:21:56 on dist-test-slave-c9qh"
server_key: "b9bc819ad3201164f33672928843cc0c"
server_key_iv: "3f38b4de95c9b30b3b3616fecc624e0e"
server_key_version: "encryptionkey@0"
I20260929 03:21:56.048810 11380 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651852476208-21193-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651852476208-21193-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651852476208-21193-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:21:56.052531 11373 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.178.125" port: 42593 } attempt: 2
I20260929 03:21:56.052934 11375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:56.052810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.125:42593 (local address 127.0.0.1:57580)
0929 03:21:56.052863 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0929 03:21:56.052905 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.178.125:42593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260929 03:21:56.053273 11375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:56.053217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.125:42593 (local address 127.0.0.1:57596)
0929 03:21:56.053239 (+ 22us) negotiation.cc:107] Waiting for socket to connect
0929 03:21:56.053259 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.178.125:42593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":12}
W20260929 03:21:56.053372 11373 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.178.125:42593: Network error: Client connection negotiation failed: client connection to 127.20.178.125:42593: connect: Connection refused (error 111)
I20260929 03:21:56.058511 11380 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:21:56.060156 11380 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:21:56.060261 11380 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:21:56.063902 11380 rpc_server.cc:307] RPC server started. Bound to: 127.20.178.125:42593
I20260929 03:21:56.063930 11447 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.178.125:42593 every 8 connection(s)
I20260929 03:21:56.064263 11380 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651852476208-21193-0/minicluster-data/master-1/data/info.pb
I20260929 03:21:56.065116 11448 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:21:56.065407 21193 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuoaHbX/build/release/bin/kudu as pid 11380
I20260929 03:21:56.065536 21193 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651852476208-21193-0/minicluster-data/master-1/wal/instance
I20260929 03:21:56.065743 21193 external_mini_cluster.cc:1428] Setting key 9396abb0f90a3b4ed91c58b8a269e626
I20260929 03:21:56.066689 11448 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.178.126" port: 39889 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 03:21:56 dist-test-slave-c9qh krb5kdc[11292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652116, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.125@KRBTEST.COM for kudu/127.20.178.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.20.178.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.178.124@KRBTEST.COM" created.
I20260929 03:21:56.075671 11376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:56.067456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48654 (local address 127.20.178.126:39889)
0929 03:21:56.067520 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0929 03:21:56.067523 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:21:56.067770 (+ 247us) server_negotiation.cc:407] Connection header received
0929 03:21:56.068329 (+ 559us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:21:56.068330 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:21:56.068342 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:21:56.068409 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:21:56.069036 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:56.069828 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:56.070746 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:56.071080 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:56.071160 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:21:56.073157 (+ 1997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:21:56.073158 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:21:56.073159 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:21:56.073167 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:21:56.074409 (+ 1242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:21:56.074785 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:21:56.074786 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:21:56.074787 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:21:56.074853 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:21:56.075070 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:21:56.075070 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:21:56.075070 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:21:56.075174 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:21:56.075225 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0929 03:21:56.075505 (+ 280us) server_negotiation.cc:299] Negotiation successful
0929 03:21:56.075603 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260929 03:21:56.075935 11451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:56.067433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:39889 (local address 127.0.0.1:48654)
0929 03:21:56.067631 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0929 03:21:56.067674 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 03:21:56.068229 (+ 555us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:21:56.068463 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:21:56.068466 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:21:56.068766 (+ 300us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:21:56.068947 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:21:56.068951 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:21:56.069918 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:21:56.069919 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:21:56.070666 (+ 747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:21:56.070668 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:21:56.071161 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:21:56.071162 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:21:56.071253 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:21:56.071736 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:21:56.071744 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:21:56.073077 (+ 1333us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:21:56.074474 (+ 1397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:21:56.074480 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:21:56.074482 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:21:56.074686 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:21:56.074915 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:21:56.074916 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:21:56.074916 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:21:56.075017 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:21:56.075227 (+ 210us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:21:56.075231 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:21:56.075358 (+ 127us) client_negotiation.cc:769] Sending connection context
0929 03:21:56.075476 (+ 118us) client_negotiation.cc:240] Negotiation successful
0929 03:21:56.075658 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":104,"threads_started":1}
I20260929 03:21:56.076436 11448 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.178.125" port: 42593 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 03:21:56 dist-test-slave-c9qh krb5kdc[11292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652116, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.125@KRBTEST.COM for kudu/127.20.178.125@KRBTEST.COM
I20260929 03:21:56.083936 11451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:56.076700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.125:42593 (local address 127.0.0.1:57606)
0929 03:21:56.076873 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0929 03:21:56.076883 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 03:21:56.076937 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:21:56.077114 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:21:56.077114 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:21:56.077267 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:21:56.077321 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:21:56.077322 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:21:56.078112 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:21:56.078113 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:21:56.078651 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:21:56.078653 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:21:56.079330 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:21:56.079331 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:21:56.079380 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:21:56.079774 (+ 394us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:21:56.079778 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:21:56.081278 (+ 1500us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:21:56.082911 (+ 1633us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:21:56.082912 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:21:56.082912 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:21:56.083072 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:21:56.083273 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:21:56.083274 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:21:56.083275 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:21:56.083325 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:21:56.083607 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:21:56.083607 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:21:56.083643 (+ 36us) client_negotiation.cc:769] Sending connection context
0929 03:21:56.083713 (+ 70us) client_negotiation.cc:240] Negotiation successful
0929 03:21:56.083829 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164}
I20260929 03:21:56.084028 11452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:56.076743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57606 (local address 127.20.178.125:42593)
0929 03:21:56.076884 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0929 03:21:56.076889 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:21:56.076965 (+ 76us) server_negotiation.cc:407] Connection header received
0929 03:21:56.076997 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:21:56.077000 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:21:56.077017 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:21:56.077102 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:21:56.077395 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:56.078040 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:56.078743 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:56.079251 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:56.079325 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:21:56.081356 (+ 2031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:21:56.081362 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:21:56.081362 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:21:56.081368 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:21:56.082800 (+ 1432us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:21:56.083159 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:21:56.083162 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:21:56.083162 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:21:56.083212 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:21:56.083391 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:21:56.083392 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:21:56.083392 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:21:56.083538 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:21:56.083627 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0929 03:21:56.083742 (+ 115us) server_negotiation.cc:299] Negotiation successful
0929 03:21:56.083894 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":87,"server-negotiator.queue_time_us":111,"thread_start_us":86,"threads_started":1}
I20260929 03:21:56.084599 11448 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.178.124" port: 37121 } has no permanent_uuid. Determining permanent_uuid...
I20260929 03:21:56.084936 11451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:56.084840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.124:37121 (local address 127.0.0.1:45506)
0929 03:21:56.084880 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0929 03:21:56.084919 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.178.124:37121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260929 03:21:56.085063 11397 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.178.124:37121: connect: Connection refused (error 111)
I20260929 03:21:56.085421 11451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:56.085333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.124:37121 (local address 127.0.0.1:45516)
0929 03:21:56.085378 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0929 03:21:56.085401 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.178.124:37121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260929 03:21:56.085563 11448 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.178.124:37121: Network error: Client connection negotiation failed: client connection to 127.20.178.124:37121: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.178.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.124.keytab.
Entry for principal kudu/127.20.178.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.124.keytab.
I20260929 03:21:56.091177 21193 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuoaHbX/build/release/bin/kudu
/tmp/dist-test-taskuoaHbX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651852476208-21193-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651852476208-21193-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651852476208-21193-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651852476208-21193-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.178.124:37121
--webserver_interface=127.20.178.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.124.keytab
--principal=kudu/127.20.178.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.178.84:35159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.178.126:39889,127.20.178.125:42593,127.20.178.124:37121
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651852476208-21193-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651852476208-21193-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651852476208-21193-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 03:21:56.134792 11448 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.178.124" port: 37121 } attempt: 1
I20260929 03:21:56.135262 11451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:56.135136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.124:37121 (local address 127.0.0.1:45526)
0929 03:21:56.135195 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0929 03:21:56.135232 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.178.124:37121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260929 03:21:56.135586 11451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:56.135498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.124:37121 (local address 127.0.0.1:45534)
0929 03:21:56.135540 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0929 03:21:56.135564 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.178.124:37121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260929 03:21:56.135664 11448 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.178.124:37121: Network error: Client connection negotiation failed: client connection to 127.20.178.124:37121: connect: Connection refused (error 111)
I20260929 03:21:56.144508 11373 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.178.125" port: 42593 } attempt: 3
Sep 29 03:21:56 dist-test-slave-c9qh krb5kdc[11292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652115, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.126@KRBTEST.COM for kudu/127.20.178.125@KRBTEST.COM
I20260929 03:21:56.151350 11452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:56.144863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57612 (local address 127.20.178.125:42593)
0929 03:21:56.144921 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0929 03:21:56.144923 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:21:56.144934 (+ 11us) server_negotiation.cc:407] Connection header received
0929 03:21:56.145000 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:21:56.145001 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:21:56.145014 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:21:56.145074 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:21:56.145383 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:56.145872 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:56.146558 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:56.146770 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:56.146850 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:21:56.148862 (+ 2012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:21:56.148864 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:21:56.148864 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:21:56.148870 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:21:56.150247 (+ 1377us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:21:56.150626 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:21:56.150627 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:21:56.150628 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:21:56.150707 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:21:56.150927 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:21:56.150928 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:21:56.150928 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:21:56.151033 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:21:56.151087 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0929 03:21:56.151236 (+ 149us) server_negotiation.cc:299] Negotiation successful
0929 03:21:56.151292 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260929 03:21:56.151418 11375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:56.144803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.125:42593 (local address 127.0.0.1:57612)
0929 03:21:56.144855 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0929 03:21:56.144864 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 03:21:56.144933 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:21:56.145093 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:21:56.145094 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:21:56.145245 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:21:56.145305 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:21:56.145306 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:21:56.145951 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:21:56.145953 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:21:56.146475 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:21:56.146477 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:21:56.146850 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:21:56.146851 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:21:56.146926 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:21:56.147368 (+ 442us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:21:56.147372 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:21:56.148779 (+ 1407us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:21:56.150325 (+ 1546us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:21:56.150326 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:21:56.150326 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:21:56.150549 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:21:56.150798 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:21:56.150799 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:21:56.150799 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:21:56.150866 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:21:56.151093 (+ 227us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:21:56.151094 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:21:56.151144 (+ 50us) client_negotiation.cc:769] Sending connection context
0929 03:21:56.151220 (+ 76us) client_negotiation.cc:240] Negotiation successful
0929 03:21:56.151315 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260929 03:21:56.151939 11373 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.178.124" port: 37121 } has no permanent_uuid. Determining permanent_uuid...
I20260929 03:21:56.152266 11375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:56.152159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.124:37121 (local address 127.0.0.1:45538)
0929 03:21:56.152203 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0929 03:21:56.152238 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.178.124:37121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260929 03:21:56.152586 11375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:56.152502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.124:37121 (local address 127.0.0.1:45552)
0929 03:21:56.152539 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0929 03:21:56.152565 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.178.124:37121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260929 03:21:56.152707 11373 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.178.124:37121: Network error: Client connection negotiation failed: client connection to 127.20.178.124:37121: connect: Connection refused (error 111)
W20260929 03:21:56.172506 11455 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 03:21:56.172762 11455 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:21:56.172796 11455 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:21:56.172833 11455 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:21:56.173995 11455 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:21:56.174038 11455 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:21:56.174052 11455 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:21:56.174101 11455 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:21:56.174124 11455 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:21:56.175614 11455 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.178.84:35159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651852476208-21193-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651852476208-21193-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.20.178.126:39889,127.20.178.125:42593,127.20.178.124:37121
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651852476208-21193-0/krb5kdc/kudu_127.20.178.124.keytab
--principal=kudu/127.20.178.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.178.124:37121
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651852476208-21193-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.20.178.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11455
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651852476208-21193-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:21:56.175850 11455 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:21:56.176069 11455 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:21:56.180703 11463 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:21:56.180675 11461 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:21:56.180685 11460 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:21:56.180779 11455 server_base.cc:1034] running on GCE node
I20260929 03:21:56.181109 11455 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:21:56.181385 11455 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:21:56.182523 11455 hybrid_clock.cc:648] HybridClock initialized: now 1790652116182511 us; error 28 us; skew 500 ppm
I20260929 03:21:56.183846 11373 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.178.124" port: 37121 } attempt: 1
I20260929 03:21:56.184177 11375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:56.184068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.124:37121 (local address 127.0.0.1:45562)
0929 03:21:56.184119 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0929 03:21:56.184146 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.178.124:37121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260929 03:21:56.184466 11375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:56.184381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.124:37121 (local address 127.0.0.1:45570)
0929 03:21:56.184418 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0929 03:21:56.184445 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.178.124:37121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
W20260929 03:21:56.184576 11373 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.178.124:37121: Network error: Client connection negotiation failed: client connection to 127.20.178.124:37121: connect: Connection refused (error 111)
Sep 29 03:21:56 dist-test-slave-c9qh krb5kdc[11292](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652116, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:21:56.185132 11455 init.cc:377] Logged in from keytab as kudu/127.20.178.124@KRBTEST.COM (short username kudu)
I20260929 03:21:56.185497 11455 webserver.cc:460] Webserver started at http://127.20.178.124:34337/ using document root <none> and password file <none>
I20260929 03:21:56.185669 11455 fs_manager.cc:359] Metadata directory not provided
I20260929 03:21:56.185707 11455 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:21:56.185791 11455 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:21:56.186731 11455 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651852476208-21193-0/minicluster-data/master-2/data/instance:
uuid: "94d47136ff5e4b118b6c7bc5fba84bae"
format_stamp: "Formatted at 2026-09-29 03:21:56 on dist-test-slave-c9qh"
server_key: "7b3e3234f0f0f365c1d6bcb59dcbf153"
server_key_iv: "4a55acf262803197e684c77b2ac02672"
server_key_version: "encryptionkey@0"
I20260929 03:21:56.187017 11455 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651852476208-21193-0/minicluster-data/master-2/wal/instance:
uuid: "94d47136ff5e4b118b6c7bc5fba84bae"
format_stamp: "Formatted at 2026-09-29 03:21:56 on dist-test-slave-c9qh"
server_key: "7b3e3234f0f0f365c1d6bcb59dcbf153"
server_key_iv: "4a55acf262803197e684c77b2ac02672"
server_key_version: "encryptionkey@0"
I20260929 03:21:56.188196 11455 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 03:21:56.188856 11470 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:21:56.189030 11455 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260929 03:21:56.189113 11455 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651852476208-21193-0/minicluster-data/master-2/data,/tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651852476208-21193-0/minicluster-data/master-2/wal
uuid: "94d47136ff5e4b118b6c7bc5fba84bae"
format_stamp: "Formatted at 2026-09-29 03:21:56 on dist-test-slave-c9qh"
server_key: "7b3e3234f0f0f365c1d6bcb59dcbf153"
server_key_iv: "4a55acf262803197e684c77b2ac02672"
server_key_version: "encryptionkey@0"
I20260929 03:21:56.189181 11455 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651852476208-21193-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651852476208-21193-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651852476208-21193-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:21:56.203785 11448 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.178.124" port: 37121 } attempt: 2
I20260929 03:21:56.204172 11451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:56.204043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.124:37121 (local address 127.0.0.1:45572)
0929 03:21:56.204097 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0929 03:21:56.204137 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.178.124:37121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260929 03:21:56.204514 11451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:56.204416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.124:37121 (local address 127.0.0.1:45582)
0929 03:21:56.204452 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0929 03:21:56.204474 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.178.124:37121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260929 03:21:56.204603 11448 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.178.124:37121: Network error: Client connection negotiation failed: client connection to 127.20.178.124:37121: connect: Connection refused (error 111)
I20260929 03:21:56.213883 11455 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:21:56.216059 11455 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:21:56.216207 11455 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:21:56.219836 11455 rpc_server.cc:307] RPC server started. Bound to: 127.20.178.124:37121
I20260929 03:21:56.219904 11522 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.178.124:37121 every 8 connection(s)
I20260929 03:21:56.220221 11455 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651852476208-21193-0/minicluster-data/master-2/data/info.pb
I20260929 03:21:56.221091 11523 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:21:56.222569 11523 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.178.126" port: 39889 } has no permanent_uuid. Determining permanent_uuid...
I20260929 03:21:56.225617 21193 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuoaHbX/build/release/bin/kudu as pid 11455
I20260929 03:21:56.225731 21193 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuoaHbX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651852476208-21193-0/minicluster-data/master-2/wal/instance
I20260929 03:21:56.225905 21193 external_mini_cluster.cc:1428] Setting key 5114181edadad94febfc969fb7e1db79
Sep 29 03:21:56 dist-test-slave-c9qh krb5kdc[11292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652116, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.124@KRBTEST.COM for kudu/127.20.178.126@KRBTEST.COM
Sep 29 03:21:56 dist-test-slave-c9qh krb5kdc[11292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652115, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.178.124@KRBTEST.COM
I20260929 03:21:56.231675 11376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:56.223208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48660 (local address 127.20.178.126:39889)
0929 03:21:56.223282 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0929 03:21:56.223285 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:21:56.223452 (+ 167us) server_negotiation.cc:407] Connection header received
0929 03:21:56.223901 (+ 449us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:21:56.223902 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:21:56.223911 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:21:56.223961 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:21:56.224483 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:56.225008 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:56.226266 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:56.226528 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:56.226584 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:21:56.228910 (+ 2326us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:21:56.228912 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:21:56.228912 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:21:56.228924 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:21:56.230286 (+ 1362us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:21:56.230680 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:21:56.230681 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:21:56.230681 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:21:56.230739 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:21:56.230994 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:21:56.230995 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:21:56.230996 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:21:56.231104 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:21:56.231176 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0929 03:21:56.231504 (+ 328us) server_negotiation.cc:299] Negotiation successful
0929 03:21:56.231588 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260929 03:21:56.231833 11525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:56.223183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.126:39889 (local address 127.0.0.1:48660)
0929 03:21:56.223345 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0929 03:21:56.223379 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0929 03:21:56.223822 (+ 443us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:21:56.224013 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:21:56.224016 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:21:56.224237 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:21:56.224415 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:21:56.224419 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:21:56.225086 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:21:56.225087 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:21:56.226159 (+ 1072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:21:56.226162 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:21:56.226589 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:21:56.226591 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:21:56.226730 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:21:56.227232 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:21:56.227240 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:21:56.228821 (+ 1581us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:21:56.230379 (+ 1558us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:21:56.230386 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:21:56.230387 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:21:56.230607 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:21:56.230802 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:21:56.230803 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:21:56.230804 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:21:56.230913 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:21:56.231168 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:21:56.231174 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:21:56.231347 (+ 173us) client_negotiation.cc:769] Sending connection context
0929 03:21:56.231464 (+ 117us) client_negotiation.cc:240] Negotiation successful
0929 03:21:56.231607 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":90,"threads_started":1}
I20260929 03:21:56.232432 11523 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.178.125" port: 42593 } has no permanent_uuid. Determining permanent_uuid...
I20260929 03:21:56.234709 11527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:56.226267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45596 (local address 127.20.178.124:37121)
0929 03:21:56.226447 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0929 03:21:56.226452 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:21:56.226470 (+ 18us) server_negotiation.cc:407] Connection header received
0929 03:21:56.226530 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:21:56.226532 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:21:56.226562 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:21:56.226643 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:21:56.227352 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:56.228241 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:56.229064 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:56.229524 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:56.229601 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:21:56.231660 (+ 2059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:21:56.231668 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:21:56.231669 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:21:56.231675 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:21:56.233678 (+ 2003us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:21:56.234004 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:21:56.234006 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:21:56.234007 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:21:56.234062 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:21:56.234248 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:21:56.234249 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:21:56.234249 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:21:56.234363 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:21:56.234473 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0929 03:21:56.234563 (+ 90us) server_negotiation.cc:299] Negotiation successful
0929 03:21:56.234617 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":84,"threads_started":1}
Sep 29 03:21:56 dist-test-slave-c9qh krb5kdc[11292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652116, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.124@KRBTEST.COM for kudu/127.20.178.125@KRBTEST.COM
I20260929 03:21:56.239554 11452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:56.232760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57618 (local address 127.20.178.125:42593)
0929 03:21:56.232812 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0929 03:21:56.232814 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:21:56.232824 (+ 10us) server_negotiation.cc:407] Connection header received
0929 03:21:56.232850 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:21:56.232851 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:21:56.232862 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:21:56.232917 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:21:56.233201 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:56.233893 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:56.234709 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:56.235011 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:56.235097 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:21:56.237025 (+ 1928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:21:56.237027 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:21:56.237027 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:21:56.237035 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:21:56.238545 (+ 1510us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:21:56.238916 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:21:56.238916 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:21:56.238917 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:21:56.238961 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:21:56.239143 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:21:56.239144 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:21:56.239144 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:21:56.239234 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:21:56.239281 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0929 03:21:56.239437 (+ 156us) server_negotiation.cc:299] Negotiation successful
0929 03:21:56.239488 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260929 03:21:56.239621 11525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:56.232700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.125:42593 (local address 127.0.0.1:57618)
0929 03:21:56.232740 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0929 03:21:56.232749 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 03:21:56.232784 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:21:56.232933 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:21:56.232934 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:21:56.233099 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:21:56.233125 (+ 26us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:21:56.233126 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:21:56.233960 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:21:56.233961 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:21:56.234631 (+ 670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:21:56.234634 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:21:56.235097 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:21:56.235098 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:21:56.235157 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:21:56.235572 (+ 415us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:21:56.235577 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:21:56.236934 (+ 1357us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:21:56.238622 (+ 1688us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:21:56.238623 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:21:56.238623 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:21:56.238843 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:21:56.239024 (+ 181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:21:56.239025 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:21:56.239025 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:21:56.239084 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:21:56.239296 (+ 212us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:21:56.239296 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:21:56.239339 (+ 43us) client_negotiation.cc:769] Sending connection context
0929 03:21:56.239423 (+ 84us) client_negotiation.cc:240] Negotiation successful
0929 03:21:56.239509 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
Sep 29 03:21:56 dist-test-slave-c9qh krb5kdc[11292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652115, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.178.126@KRBTEST.COM
I20260929 03:21:56.239941 11523 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.178.124" port: 37121 } has no permanent_uuid. Determining permanent_uuid...
I20260929 03:21:56.243146 11376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:56.235423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48662 (local address 127.20.178.126:39889)
0929 03:21:56.235460 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0929 03:21:56.235462 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:21:56.235628 (+ 166us) server_negotiation.cc:407] Connection header received
0929 03:21:56.235686 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:21:56.235687 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:21:56.235694 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:21:56.235732 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:21:56.236267 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:56.237021 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:56.237655 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:56.237847 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:56.237923 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:21:56.240315 (+ 2392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:21:56.240316 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:21:56.240316 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:21:56.240322 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:21:56.241740 (+ 1418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:21:56.242095 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:21:56.242096 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:21:56.242096 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:21:56.242150 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:21:56.242308 (+ 158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:21:56.242309 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:21:56.242309 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:21:56.242439 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:21:56.242904 (+ 465us) server_negotiation.cc:1035] Waiting for connection context
0929 03:21:56.243031 (+ 127us) server_negotiation.cc:299] Negotiation successful
0929 03:21:56.243090 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
Sep 29 03:21:56 dist-test-slave-c9qh krb5kdc[11292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652116, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.124@KRBTEST.COM for kudu/127.20.178.124@KRBTEST.COM
I20260929 03:21:56.246367 11525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:56.240168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.124:37121 (local address 127.0.0.1:45602)
0929 03:21:56.240256 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0929 03:21:56.240265 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 03:21:56.240309 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:21:56.240421 (+ 112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:21:56.240422 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:21:56.240524 (+ 102us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:21:56.240547 (+ 23us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:21:56.240548 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:21:56.241160 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:21:56.241161 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:21:56.241603 (+ 442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:21:56.241604 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:21:56.242098 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:21:56.242099 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:21:56.242160 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:21:56.242494 (+ 334us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:21:56.242498 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:21:56.243752 (+ 1254us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:21:56.245445 (+ 1693us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:21:56.245446 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:21:56.245446 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:21:56.245604 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:21:56.245805 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:21:56.245806 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:21:56.245806 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:21:56.245850 (+ 44us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:21:56.246059 (+ 209us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:21:56.246059 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:21:56.246088 (+ 29us) client_negotiation.cc:769] Sending connection context
0929 03:21:56.246165 (+ 77us) client_negotiation.cc:240] Negotiation successful
0929 03:21:56.246251 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":75}
I20260929 03:21:56.246376 11527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:56.240216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45602 (local address 127.20.178.124:37121)
0929 03:21:56.240253 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0929 03:21:56.240254 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 03:21:56.240296 (+ 42us) server_negotiation.cc:407] Connection header received
0929 03:21:56.240355 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:21:56.240356 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:21:56.240365 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:21:56.240413 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:21:56.240615 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:56.241077 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:56.241674 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:56.242027 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:56.242090 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:21:56.243867 (+ 1777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:21:56.243868 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:21:56.243868 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:21:56.243874 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:21:56.245370 (+ 1496us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:21:56.245667 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:21:56.245668 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:21:56.245668 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:21:56.245725 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:21:56.245889 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:21:56.245889 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:21:56.245890 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:21:56.246005 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:21:56.246068 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0929 03:21:56.246203 (+ 135us) server_negotiation.cc:299] Negotiation successful
0929 03:21:56.246295 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":32,"server-negotiator.queue_time_us":18}
I20260929 03:21:56.247597 11523 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94d47136ff5e4b118b6c7bc5fba84bae: Bootstrap starting.
Sep 29 03:21:56 dist-test-slave-c9qh krb5kdc[11292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652115, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.178.125@KRBTEST.COM
I20260929 03:21:56.248602 11523 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 94d47136ff5e4b118b6c7bc5fba84bae: Neither blocks nor log segments found. Creating new log.
I20260929 03:21:56.248950 11523 log.cc:824] T 00000000000000000000000000000000 P 94d47136ff5e4b118b6c7bc5fba84bae: Log is configured to *not* fsync() on all Append() calls
I20260929 03:21:56.249675 11523 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94d47136ff5e4b118b6c7bc5fba84bae: No bootstrap required, opened a new log
I20260929 03:21:56.251219 11523 raft_consensus.cc:348] T 00000000000000000000000000000000 P 94d47136ff5e4b118b6c7bc5fba84bae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "204339ee7e414eed9135250b17101d3b" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 39889 } } peers { permanent_uuid: "b2f4204b0ccd4581a2b7838e1458d704" member_type: VOTER last_known_addr { host: "127.20.178.125" port: 42593 } } peers { permanent_uuid: "94d47136ff5e4b118b6c7bc5fba84bae" member_type: VOTER last_known_addr { host: "127.20.178.124" port: 37121 } }
I20260929 03:21:56.251327 11523 raft_consensus.cc:729] T 00000000000000000000000000000000 P 94d47136ff5e4b118b6c7bc5fba84bae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94d47136ff5e4b118b6c7bc5fba84bae, State: Initialized, Role: FOLLOWER
I20260929 03:21:56.251379 11523 consensus_queue.cc:260] T 00000000000000000000000000000000 P 94d47136ff5e4b118b6c7bc5fba84bae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "204339ee7e414eed9135250b17101d3b" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 39889 } } peers { permanent_uuid: "b2f4204b0ccd4581a2b7838e1458d704" member_type: VOTER last_known_addr { host: "127.20.178.125" port: 42593 } } peers { permanent_uuid: "94d47136ff5e4b118b6c7bc5fba84bae" member_type: VOTER last_known_addr { host: "127.20.178.124" port: 37121 } }
I20260929 03:21:56.251564 11452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:56.244019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57628 (local address 127.20.178.125:42593)
0929 03:21:56.244055 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0929 03:21:56.244057 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:21:56.244065 (+ 8us) server_negotiation.cc:407] Connection header received
0929 03:21:56.244124 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:21:56.244125 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:21:56.244134 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:21:56.244197 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:21:56.244841 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:56.245287 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:56.246138 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:56.246533 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:56.246603 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:21:56.249052 (+ 2449us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:21:56.249054 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:21:56.249054 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:21:56.249062 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:21:56.250422 (+ 1360us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:21:56.250805 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:21:56.250805 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:21:56.250806 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:21:56.250850 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:21:56.251072 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:21:56.251073 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:21:56.251073 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:21:56.251188 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:21:56.251363 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0929 03:21:56.251398 (+ 35us) server_negotiation.cc:299] Negotiation successful
0929 03:21:56.251463 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260929 03:21:56.251657 11523 sys_catalog.cc:564] T 00000000000000000000000000000000 P 94d47136ff5e4b118b6c7bc5fba84bae [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:21:56.251611 11529 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94d47136ff5e4b118b6c7bc5fba84bae [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "204339ee7e414eed9135250b17101d3b" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 39889 } } peers { permanent_uuid: "b2f4204b0ccd4581a2b7838e1458d704" member_type: VOTER last_known_addr { host: "127.20.178.125" port: 42593 } } peers { permanent_uuid: "94d47136ff5e4b118b6c7bc5fba84bae" member_type: VOTER last_known_addr { host: "127.20.178.124" port: 37121 } } }
I20260929 03:21:56.251721 11529 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94d47136ff5e4b118b6c7bc5fba84bae [sys.catalog]: This master's current role is: FOLLOWER
I20260929 03:21:56.253073 21193 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260929 03:21:56.253466 11540 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 94d47136ff5e4b118b6c7bc5fba84bae: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 03:21:56.253511 11540 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 03:21:56.253971 11347 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "204339ee7e414eed9135250b17101d3b"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48662
I20260929 03:21:56.253724 11455 master_runner.cc:186] Error getting master registration for 127.20.178.126:39889: OK, instance_id { permanent_uuid: "204339ee7e414eed9135250b17101d3b" instance_seqno: 1790652115918624 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 03:21:56.255745 11455 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 03:21:56.259666 11373 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.178.124" port: 37121 } attempt: 2
Sep 29 03:21:56 dist-test-slave-c9qh krb5kdc[11292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652115, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.126@KRBTEST.COM for kudu/127.20.178.124@KRBTEST.COM
I20260929 03:21:56.266989 11375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:56.259893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.124:37121 (local address 127.0.0.1:45604)
0929 03:21:56.259970 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0929 03:21:56.259982 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 03:21:56.260020 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:21:56.260148 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:21:56.260149 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:21:56.260268 (+ 119us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:21:56.260300 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:21:56.260301 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:21:56.261196 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:21:56.261197 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:21:56.261984 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:21:56.261987 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:21:56.262446 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:21:56.262447 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:21:56.262509 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:21:56.262957 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:21:56.262961 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:21:56.264163 (+ 1202us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:21:56.265913 (+ 1750us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:21:56.265914 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:21:56.265914 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:21:56.266135 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:21:56.266338 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:21:56.266339 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:21:56.266339 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:21:56.266397 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:21:56.266686 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:21:56.266688 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:21:56.266732 (+ 44us) client_negotiation.cc:769] Sending connection context
0929 03:21:56.266798 (+ 66us) client_negotiation.cc:240] Negotiation successful
0929 03:21:56.266884 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61}
I20260929 03:21:56.267001 11527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:56.259992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45604 (local address 127.20.178.124:37121)
0929 03:21:56.260025 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0929 03:21:56.260027 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:21:56.260060 (+ 33us) server_negotiation.cc:407] Connection header received
0929 03:21:56.260078 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:21:56.260079 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:21:56.260088 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:21:56.260139 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:21:56.260378 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:56.261125 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:56.262070 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:56.262361 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:56.262427 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:21:56.264262 (+ 1835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:21:56.264263 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:21:56.264264 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:21:56.264271 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:21:56.265839 (+ 1568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:21:56.266213 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:21:56.266214 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:21:56.266214 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:21:56.266272 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:21:56.266458 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:21:56.266458 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:21:56.266459 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:21:56.266621 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:21:56.266690 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0929 03:21:56.266830 (+ 140us) server_negotiation.cc:299] Negotiation successful
0929 03:21:56.266904 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260929 03:21:56.268579 11373 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 204339ee7e414eed9135250b17101d3b: Bootstrap starting.
I20260929 03:21:56.269284 11373 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 204339ee7e414eed9135250b17101d3b: Neither blocks nor log segments found. Creating new log.
I20260929 03:21:56.269562 11373 log.cc:824] T 00000000000000000000000000000000 P 204339ee7e414eed9135250b17101d3b: Log is configured to *not* fsync() on all Append() calls
I20260929 03:21:56.270272 11373 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 204339ee7e414eed9135250b17101d3b: No bootstrap required, opened a new log
I20260929 03:21:56.271260 11373 raft_consensus.cc:348] T 00000000000000000000000000000000 P 204339ee7e414eed9135250b17101d3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "204339ee7e414eed9135250b17101d3b" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 39889 } } peers { permanent_uuid: "b2f4204b0ccd4581a2b7838e1458d704" member_type: VOTER last_known_addr { host: "127.20.178.125" port: 42593 } } peers { permanent_uuid: "94d47136ff5e4b118b6c7bc5fba84bae" member_type: VOTER last_known_addr { host: "127.20.178.124" port: 37121 } }
I20260929 03:21:56.271387 11373 raft_consensus.cc:729] T 00000000000000000000000000000000 P 204339ee7e414eed9135250b17101d3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 204339ee7e414eed9135250b17101d3b, State: Initialized, Role: FOLLOWER
I20260929 03:21:56.271478 11373 consensus_queue.cc:260] T 00000000000000000000000000000000 P 204339ee7e414eed9135250b17101d3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "204339ee7e414eed9135250b17101d3b" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 39889 } } peers { permanent_uuid: "b2f4204b0ccd4581a2b7838e1458d704" member_type: VOTER last_known_addr { host: "127.20.178.125" port: 42593 } } peers { permanent_uuid: "94d47136ff5e4b118b6c7bc5fba84bae" member_type: VOTER last_known_addr { host: "127.20.178.124" port: 37121 } }
I20260929 03:21:56.271754 11543 sys_catalog.cc:455] T 00000000000000000000000000000000 P 204339ee7e414eed9135250b17101d3b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "204339ee7e414eed9135250b17101d3b" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 39889 } } peers { permanent_uuid: "b2f4204b0ccd4581a2b7838e1458d704" member_type: VOTER last_known_addr { host: "127.20.178.125" port: 42593 } } peers { permanent_uuid: "94d47136ff5e4b118b6c7bc5fba84bae" member_type: VOTER last_known_addr { host: "127.20.178.124" port: 37121 } } }
I20260929 03:21:56.271847 11543 sys_catalog.cc:458] T 00000000000000000000000000000000 P 204339ee7e414eed9135250b17101d3b [sys.catalog]: This master's current role is: FOLLOWER
I20260929 03:21:56.271859 11347 raft_consensus.cc:482] T 00000000000000000000000000000000 P 204339ee7e414eed9135250b17101d3b [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260929 03:21:56.271908 11347 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 204339ee7e414eed9135250b17101d3b [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:21:56.272437 11347 raft_consensus.cc:504] T 00000000000000000000000000000000 P 204339ee7e414eed9135250b17101d3b [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "204339ee7e414eed9135250b17101d3b" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 39889 } } peers { permanent_uuid: "b2f4204b0ccd4581a2b7838e1458d704" member_type: VOTER last_known_addr { host: "127.20.178.125" port: 42593 } } peers { permanent_uuid: "94d47136ff5e4b118b6c7bc5fba84bae" member_type: VOTER last_known_addr { host: "127.20.178.124" port: 37121 } }
I20260929 03:21:56.272667 11347 leader_election.cc:288] T 00000000000000000000000000000000 P 204339ee7e414eed9135250b17101d3b [CANDIDATE]: Term 1 election: Requested vote from peers b2f4204b0ccd4581a2b7838e1458d704 (127.20.178.125:42593), 94d47136ff5e4b118b6c7bc5fba84bae (127.20.178.124:37121)
I20260929 03:21:56.271799 11373 sys_catalog.cc:564] T 00000000000000000000000000000000 P 204339ee7e414eed9135250b17101d3b [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:21:56.273058 11497 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "204339ee7e414eed9135250b17101d3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "94d47136ff5e4b118b6c7bc5fba84bae"
I20260929 03:21:56.273173 11497 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 94d47136ff5e4b118b6c7bc5fba84bae [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:21:56.274178 11497 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 94d47136ff5e4b118b6c7bc5fba84bae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 204339ee7e414eed9135250b17101d3b in term 1.
I20260929 03:21:56.274405 11322 leader_election.cc:302] T 00000000000000000000000000000000 P 204339ee7e414eed9135250b17101d3b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 204339ee7e414eed9135250b17101d3b, 94d47136ff5e4b118b6c7bc5fba84bae; no voters:
I20260929 03:21:56.274519 11543 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 204339ee7e414eed9135250b17101d3b [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:21:56.274637 11543 raft_consensus.cc:686] T 00000000000000000000000000000000 P 204339ee7e414eed9135250b17101d3b [term 1 LEADER]: Becoming Leader. State: Replica: 204339ee7e414eed9135250b17101d3b, State: Running, Role: LEADER
I20260929 03:21:56.274711 11543 consensus_queue.cc:237] T 00000000000000000000000000000000 P 204339ee7e414eed9135250b17101d3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "204339ee7e414eed9135250b17101d3b" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 39889 } } peers { permanent_uuid: "b2f4204b0ccd4581a2b7838e1458d704" member_type: VOTER last_known_addr { host: "127.20.178.125" port: 42593 } } peers { permanent_uuid: "94d47136ff5e4b118b6c7bc5fba84bae" member_type: VOTER last_known_addr { host: "127.20.178.124" port: 37121 } }
I20260929 03:21:56.275110 11543 sys_catalog.cc:455] T 00000000000000000000000000000000 P 204339ee7e414eed9135250b17101d3b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 204339ee7e414eed9135250b17101d3b. Latest consensus state: current_term: 1 leader_uuid: "204339ee7e414eed9135250b17101d3b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "204339ee7e414eed9135250b17101d3b" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 39889 } } peers { permanent_uuid: "b2f4204b0ccd4581a2b7838e1458d704" member_type: VOTER last_known_addr { host: "127.20.178.125" port: 42593 } } peers { permanent_uuid: "94d47136ff5e4b118b6c7bc5fba84bae" member_type: VOTER last_known_addr { host: "127.20.178.124" port: 37121 } } }
I20260929 03:21:56.275177 11543 sys_catalog.cc:458] T 00000000000000000000000000000000 P 204339ee7e414eed9135250b17101d3b [sys.catalog]: This master's current role is: LEADER
I20260929 03:21:56.277031 11305 master_runner.cc:186] Error getting master registration for 127.20.178.125:42593: OK, instance_id { permanent_uuid: "b2f4204b0ccd4581a2b7838e1458d704" instance_seqno: 1790652116062927 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 03:21:56.279769 11305 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 03:21:56.279954 11562 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:21:56.282845 11452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:56.273762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57630 (local address 127.20.178.125:42593)
0929 03:21:56.273803 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0929 03:21:56.273805 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:21:56.274082 (+ 277us) server_negotiation.cc:407] Connection header received
0929 03:21:56.274193 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:21:56.274195 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:21:56.274206 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:21:56.274238 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:21:56.275039 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:56.275808 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:56.277064 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:56.277795 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:56.277876 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:21:56.279521 (+ 1645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:21:56.279523 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:21:56.279523 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:21:56.279531 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:21:56.281311 (+ 1780us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:21:56.281743 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:21:56.281745 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:21:56.281745 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:21:56.281811 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:21:56.282035 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:21:56.282035 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:21:56.282036 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:21:56.282169 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:21:56.282255 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0929 03:21:56.282666 (+ 411us) server_negotiation.cc:299] Negotiation successful
0929 03:21:56.282752 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260929 03:21:56.283578 11562 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:21:56.285064 11376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:56.275403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48678 (local address 127.20.178.126:39889)
0929 03:21:56.275435 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0929 03:21:56.275437 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:21:56.275444 (+ 7us) server_negotiation.cc:407] Connection header received
0929 03:21:56.275462 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:21:56.275463 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:21:56.275473 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:21:56.275536 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:21:56.276196 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:56.276979 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:56.278034 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:56.278642 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:56.278685 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:21:56.281087 (+ 2402us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:21:56.281088 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:21:56.281088 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:21:56.281095 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:21:56.282660 (+ 1565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:21:56.283037 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:21:56.283038 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:21:56.283038 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:21:56.283089 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:21:56.284430 (+ 1341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:21:56.284431 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:21:56.284431 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:21:56.284566 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:21:56.284661 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0929 03:21:56.284901 (+ 240us) server_negotiation.cc:299] Negotiation successful
0929 03:21:56.284970 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260929 03:21:56.285738 11497 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 94d47136ff5e4b118b6c7bc5fba84bae [term 1 FOLLOWER]: Refusing update from remote peer 204339ee7e414eed9135250b17101d3b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:21:56.286135 11543 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 204339ee7e414eed9135250b17101d3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "94d47136ff5e4b118b6c7bc5fba84bae" member_type: VOTER last_known_addr { host: "127.20.178.124" port: 37121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:21:56.288045 11529 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94d47136ff5e4b118b6c7bc5fba84bae [sys.catalog]: SysCatalogTable state changed. Reason: New leader 204339ee7e414eed9135250b17101d3b. Latest consensus state: current_term: 1 leader_uuid: "204339ee7e414eed9135250b17101d3b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "204339ee7e414eed9135250b17101d3b" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 39889 } } peers { permanent_uuid: "b2f4204b0ccd4581a2b7838e1458d704" member_type: VOTER last_known_addr { host: "127.20.178.125" port: 42593 } } peers { permanent_uuid: "94d47136ff5e4b118b6c7bc5fba84bae" member_type: VOTER last_known_addr { host: "127.20.178.124" port: 37121 } } }
I20260929 03:21:56.288139 11529 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94d47136ff5e4b118b6c7bc5fba84bae [sys.catalog]: This master's current role is: FOLLOWER
I20260929 03:21:56.288046 11527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:56.275370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45618 (local address 127.20.178.124:37121)
0929 03:21:56.275686 (+ 316us) server_negotiation.cc:206] Beginning negotiation
0929 03:21:56.275688 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:21:56.279793 (+ 4105us) server_negotiation.cc:407] Connection header received
0929 03:21:56.280435 (+ 642us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:21:56.280436 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:21:56.280446 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:21:56.280483 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:21:56.281047 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:56.282306 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:56.283412 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:56.283676 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:56.283717 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:21:56.285134 (+ 1417us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:21:56.285135 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:21:56.285136 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:21:56.285143 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:21:56.286842 (+ 1699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:21:56.287221 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:21:56.287222 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:21:56.287222 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:21:56.287276 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:21:56.287498 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:21:56.287499 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:21:56.287499 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:21:56.287615 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:21:56.287688 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 03:21:56.287880 (+ 192us) server_negotiation.cc:299] Negotiation successful
0929 03:21:56.287954 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298}
I20260929 03:21:56.289825 11558 sys_catalog.cc:455] T 00000000000000000000000000000000 P 204339ee7e414eed9135250b17101d3b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "204339ee7e414eed9135250b17101d3b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "204339ee7e414eed9135250b17101d3b" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 39889 } } peers { permanent_uuid: "b2f4204b0ccd4581a2b7838e1458d704" member_type: VOTER last_known_addr { host: "127.20.178.125" port: 42593 } } peers { permanent_uuid: "94d47136ff5e4b118b6c7bc5fba84bae" member_type: VOTER last_known_addr { host: "127.20.178.124" port: 37121 } } }
I20260929 03:21:56.289918 11558 sys_catalog.cc:458] T 00000000000000000000000000000000 P 204339ee7e414eed9135250b17101d3b [sys.catalog]: This master's current role is: LEADER
I20260929 03:21:56.290166 11562 catalog_manager.cc:1347] Generated new cluster ID: b27af727351241bfa0570d81f9c38165
I20260929 03:21:56.290359 11529 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94d47136ff5e4b118b6c7bc5fba84bae [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "204339ee7e414eed9135250b17101d3b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "204339ee7e414eed9135250b17101d3b" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 39889 } } peers { permanent_uuid: "b2f4204b0ccd4581a2b7838e1458d704" member_type: VOTER last_known_addr { host: "127.20.178.125" port: 42593 } } peers { permanent_uuid: "94d47136ff5e4b118b6c7bc5fba84bae" member_type: VOTER last_known_addr { host: "127.20.178.124" port: 37121 } } }
I20260929 03:21:56.290428 11529 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94d47136ff5e4b118b6c7bc5fba84bae [sys.catalog]: This master's current role is: FOLLOWER
I20260929 03:21:56.290315 11562 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:21:56.294478 11567 mvcc.cc:205] Tried to move back new op lower bound from 7334511068302417920 to 7334511068261941248. Current Snapshot: MvccSnapshot[applied={T|T < 7334511068302417920}]
I20260929 03:21:56.295746 11448 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.178.124" port: 37121 } attempt: 3
Sep 29 03:21:56 dist-test-slave-c9qh krb5kdc[11292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652116, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.178.125@KRBTEST.COM for kudu/127.20.178.124@KRBTEST.COM
I20260929 03:21:56.303213 11527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:56.296016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45634 (local address 127.20.178.124:37121)
0929 03:21:56.296051 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0929 03:21:56.296053 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:21:56.296062 (+ 9us) server_negotiation.cc:407] Connection header received
0929 03:21:56.296141 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:21:56.296141 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:21:56.296149 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:21:56.296186 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:21:56.296472 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:56.297189 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:56.298123 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:56.298422 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:56.298474 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:21:56.300627 (+ 2153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:21:56.300628 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:21:56.300628 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:21:56.300652 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:21:56.302120 (+ 1468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:21:56.302497 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:21:56.302498 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:21:56.302499 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:21:56.302549 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:21:56.302742 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:21:56.302742 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:21:56.302743 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:21:56.302856 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:21:56.302911 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0929 03:21:56.303068 (+ 157us) server_negotiation.cc:299] Negotiation successful
0929 03:21:56.303131 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260929 03:21:56.303234 11451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:56.295972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.178.124:37121 (local address 127.0.0.1:45634)
0929 03:21:56.296015 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0929 03:21:56.296026 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 03:21:56.296078 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:21:56.296217 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:21:56.296218 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:21:56.296369 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:21:56.296407 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:21:56.296409 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:21:56.297262 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:21:56.297263 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:21:56.298036 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:21:56.298039 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:21:56.298521 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:21:56.298522 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:21:56.298577 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:21:56.299024 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:21:56.299028 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:21:56.300555 (+ 1527us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:21:56.302207 (+ 1652us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:21:56.302208 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:21:56.302209 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:21:56.302418 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:21:56.302610 (+ 192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:21:56.302611 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:21:56.302611 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:21:56.302685 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:21:56.302915 (+ 230us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:21:56.302915 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:21:56.302975 (+ 60us) client_negotiation.cc:769] Sending connection context
0929 03:21:56.303047 (+ 72us) client_negotiation.cc:240] Negotiation successful
0929 03:21:56.303137 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23}
I20260929 03:21:56.304744 11448 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2f4204b0ccd4581a2b7838e1458d704: Bootstrap starting.
I20260929 03:21:56.305464 11448 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b2f4204b0ccd4581a2b7838e1458d704: Neither blocks nor log segments found. Creating new log.
I20260929 03:21:56.305755 11448 log.cc:824] T 00000000000000000000000000000000 P b2f4204b0ccd4581a2b7838e1458d704: Log is configured to *not* fsync() on all Append() calls
I20260929 03:21:56.306697 11448 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2f4204b0ccd4581a2b7838e1458d704: No bootstrap required, opened a new log
I20260929 03:21:56.308786 11448 raft_consensus.cc:348] T 00000000000000000000000000000000 P b2f4204b0ccd4581a2b7838e1458d704 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "204339ee7e414eed9135250b17101d3b" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 39889 } } peers { permanent_uuid: "b2f4204b0ccd4581a2b7838e1458d704" member_type: VOTER last_known_addr { host: "127.20.178.125" port: 42593 } } peers { permanent_uuid: "94d47136ff5e4b118b6c7bc5fba84bae" member_type: VOTER last_known_addr { host: "127.20.178.124" port: 37121 } }
I20260929 03:21:56.309073 11448 raft_consensus.cc:729] T 00000000000000000000000000000000 P b2f4204b0ccd4581a2b7838e1458d704 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2f4204b0ccd4581a2b7838e1458d704, State: Initialized, Role: FOLLOWER
I20260929 03:21:56.309274 11448 consensus_queue.cc:260] T 00000000000000000000000000000000 P b2f4204b0ccd4581a2b7838e1458d704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "204339ee7e414eed9135250b17101d3b" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 39889 } } peers { permanent_uuid: "b2f4204b0ccd4581a2b7838e1458d704" member_type: VOTER last_known_addr { host: "127.20.178.125" port: 42593 } } peers { permanent_uuid: "94d47136ff5e4b118b6c7bc5fba84bae" member_type: VOTER last_known_addr { host: "127.20.178.124" port: 37121 } }
I20260929 03:21:56.309895 11422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "204339ee7e414eed9135250b17101d3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "b2f4204b0ccd4581a2b7838e1458d704"
I20260929 03:21:56.309995 11422 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b2f4204b0ccd4581a2b7838e1458d704 [term 0 FOLLOWER]: Advancing to term 1
W20260929 03:21:56.310746 11401 tablet.cc:2369] T 00000000000000000000000000000000 P b2f4204b0ccd4581a2b7838e1458d704: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 03:21:56.310775 11422 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b2f4204b0ccd4581a2b7838e1458d704 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 204339ee7e414eed9135250b17101d3b in term 1.
I20260929 03:21:56.311153 11421 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b2f4204b0ccd4581a2b7838e1458d704 [term 1 FOLLOWER]: Refusing update from remote peer 204339ee7e414eed9135250b17101d3b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:21:56.311412 11543 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 204339ee7e414eed9135250b17101d3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2f4204b0ccd4581a2b7838e1458d704" member_type: VOTER last_known_addr { host: "127.20.178.125" port: 42593 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:21:56.312908 11448 sys_catalog.cc:564] T 00000000000000000000000000000000 P b2f4204b0ccd4581a2b7838e1458d704 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:21:56.314311 11572 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2f4204b0ccd4581a2b7838e1458d704 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "204339ee7e414eed9135250b17101d3b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "204339ee7e414eed9135250b17101d3b" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 39889 } } peers { permanent_uuid: "b2f4204b0ccd4581a2b7838e1458d704" member_type: VOTER last_known_addr { host: "127.20.178.125" port: 42593 } } peers { permanent_uuid: "94d47136ff5e4b118b6c7bc5fba84bae" member_type: VOTER last_known_addr { host: "127.20.178.124" port: 37121 } } }
I20260929 03:21:56.314471 11572 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2f4204b0ccd4581a2b7838e1458d704 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 03:21:56.314581 11572 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2f4204b0ccd4581a2b7838e1458d704 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 204339ee7e414eed9135250b17101d3b. Latest consensus state: current_term: 1 leader_uuid: "204339ee7e414eed9135250b17101d3b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "204339ee7e414eed9135250b17101d3b" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 39889 } } peers { permanent_uuid: "b2f4204b0ccd4581a2b7838e1458d704" member_type: VOTER last_known_addr { host: "127.20.178.125" port: 42593 } } peers { permanent_uuid: "94d47136ff5e4b118b6c7bc5fba84bae" member_type: VOTER last_known_addr { host: "127.20.178.124" port: 37121 } } }
I20260929 03:21:56.314692 11572 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2f4204b0ccd4581a2b7838e1458d704 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 03:21:56.314766 11572 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2f4204b0ccd4581a2b7838e1458d704 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "204339ee7e414eed9135250b17101d3b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "204339ee7e414eed9135250b17101d3b" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 39889 } } peers { permanent_uuid: "b2f4204b0ccd4581a2b7838e1458d704" member_type: VOTER last_known_addr { host: "127.20.178.125" port: 42593 } } peers { permanent_uuid: "94d47136ff5e4b118b6c7bc5fba84bae" member_type: VOTER last_known_addr { host: "127.20.178.124" port: 37121 } } }
I20260929 03:21:56.314859 11572 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2f4204b0ccd4581a2b7838e1458d704 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 03:21:56.319464 11574 mvcc.cc:205] Tried to move back new op lower bound from 7334511068302417920 to 7334511068261941248. Current Snapshot: MvccSnapshot[applied={T|T < 7334511068302417920}]
I20260929 03:21:56.321529 11587 catalog_manager.cc:1259] Loaded cluster ID: b27af727351241bfa0570d81f9c38165
I20260929 03:21:56.321612 11587 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b2f4204b0ccd4581a2b7838e1458d704: loading cluster ID for follower catalog manager: success
W20260929 03:21:56.321734 11587 catalog_manager.cc:1576] T 00000000000000000000000000000000 P b2f4204b0ccd4581a2b7838e1458d704: acquiring CA information for follower catalog manager: Not found: root CA entry not found
W20260929 03:21:56.321810 11587 catalog_manager.cc:873] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry
I20260929 03:21:56.322096 11543 sys_catalog.cc:455] T 00000000000000000000000000000000 P 204339ee7e414eed9135250b17101d3b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "204339ee7e414eed9135250b17101d3b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "204339ee7e414eed9135250b17101d3b" member_type: VOTER last_known_addr { host: "127.20.178.126" port: 39889 } } peers { permanent_uuid: "b2f4204b0ccd4581a2b7838e1458d704" member_type: VOTER last_known_addr { host: "127.20.178.125" port: 42593 } } peers { permanent_uuid: "94d47136ff5e4b118b6c7bc5fba84bae" member_type: VOTER last_known_addr { host: "127.20.178.124" port: 37121 } } }
I20260929 03:21:56.322187 11543 sys_catalog.cc:458] T 00000000000000000000000000000000 P 204339ee7e414eed9135250b17101d3b [sys.catalog]: This master's current role is: LEADER
I20260929 03:21:56.323391 11562 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:21:56.324146 11562 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:21:56.331207 11562 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 204339ee7e414eed9135250b17101d3b: Generated new TSK 0
I20260929 03:21:56.331372 11562 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 03:21:56.356243 11452 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:21:56.354680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57636 (local address 127.20.178.125:42593)
0929 03:21:56.354729 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0929 03:21:56.354731 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:21:56.355588 (+ 857us) server_negotiation.cc:407] Connection header received
0929 03:21:56.355696 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:21:56.355697 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:21:56.355707 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:21:56.355738 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:21:56.356174 (+ 436us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57636: BlockingRecv error: recv got EOF from 127.0.0.1:57636 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
W20260929 03:21:56.356978 11527 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:21:56.355053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45638 (local address 127.20.178.124:37121)
0929 03:21:56.355100 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0929 03:21:56.355102 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:21:56.356475 (+ 1373us) server_negotiation.cc:407] Connection header received
0929 03:21:56.356562 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:21:56.356563 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:21:56.356573 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:21:56.356602 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:21:56.356908 (+ 306us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45638: BlockingRecv error: recv got EOF from 127.0.0.1:45638 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260929 03:21:56.357661 11376 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:21:56.355338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48684 (local address 127.20.178.126:39889)
0929 03:21:56.355376 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0929 03:21:56.355378 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:21:56.357184 (+ 1806us) server_negotiation.cc:407] Connection header received
0929 03:21:56.357278 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:21:56.357279 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:21:56.357290 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:21:56.357319 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:21:56.357599 (+ 280us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48684: BlockingRecv error: recv got EOF from 127.0.0.1:48684 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260929 03:21:56.359918 11452 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:21:56.358874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57652 (local address 127.20.178.125:42593)
0929 03:21:56.358913 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0929 03:21:56.358915 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:21:56.359420 (+ 505us) server_negotiation.cc:407] Connection header received
0929 03:21:56.359526 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:21:56.359527 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:21:56.359544 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:21:56.359573 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:21:56.359873 (+ 300us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57652: BlockingRecv error: recv got EOF from 127.0.0.1:57652 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260929 03:21:56.362725 11452 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:21:56.362021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57668 (local address 127.20.178.125:42593)
0929 03:21:56.362052 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0929 03:21:56.362053 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 03:21:56.362277 (+ 224us) server_negotiation.cc:407] Connection header received
0929 03:21:56.362366 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:21:56.362367 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:21:56.362375 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:21:56.362403 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:21:56.362684 (+ 281us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57668: BlockingRecv error: recv got EOF from 127.0.0.1:57668 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260929 03:21:56.366472 11452 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:21:56.365660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57670 (local address 127.20.178.125:42593)
0929 03:21:56.365690 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0929 03:21:56.365691 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 03:21:56.365905 (+ 214us) server_negotiation.cc:407] Connection header received
0929 03:21:56.366010 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:21:56.366011 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:21:56.366020 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:21:56.366062 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:21:56.366431 (+ 369us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57670: BlockingRecv error: recv got EOF from 127.0.0.1:57670 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
W20260929 03:21:56.372195 11452 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:21:56.371440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57674 (local address 127.20.178.125:42593)
0929 03:21:56.371471 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0929 03:21:56.371473 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:21:56.371732 (+ 259us) server_negotiation.cc:407] Connection header received
0929 03:21:56.371823 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:21:56.371824 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:21:56.371833 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:21:56.371860 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:21:56.372150 (+ 290us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57674: BlockingRecv error: recv got EOF from 127.0.0.1:57674 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260929 03:21:56.381963 11452 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:21:56.381220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57680 (local address 127.20.178.125:42593)
0929 03:21:56.381251 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0929 03:21:56.381252 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 03:21:56.381476 (+ 224us) server_negotiation.cc:407] Connection header received
0929 03:21:56.381570 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:21:56.381571 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:21:56.381580 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:21:56.381607 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:21:56.381895 (+ 288us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57680: BlockingRecv error: recv got EOF from 127.0.0.1:57680 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260929 03:21:56.399829 11452 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:21:56.399084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57684 (local address 127.20.178.125:42593)
0929 03:21:56.399128 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0929 03:21:56.399130 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:21:56.399358 (+ 228us) server_negotiation.cc:407] Connection header received
0929 03:21:56.399446 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:21:56.399447 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:21:56.399455 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:21:56.399483 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:21:56.399776 (+ 293us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57684: BlockingRecv error: recv got EOF from 127.0.0.1:57684 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
W20260929 03:21:56.434156 11452 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:21:56.433194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57700 (local address 127.20.178.125:42593)
0929 03:21:56.433237 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0929 03:21:56.433239 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:21:56.433554 (+ 315us) server_negotiation.cc:407] Connection header received
0929 03:21:56.433665 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:21:56.433666 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:21:56.433678 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:21:56.433708 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:21:56.434099 (+ 391us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57700: BlockingRecv error: recv got EOF from 127.0.0.1:57700 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260929 03:21:56.500036 11452 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:21:56.499409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57702 (local address 127.20.178.125:42593)
0929 03:21:56.499455 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0929 03:21:56.499458 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:21:56.499585 (+ 127us) server_negotiation.cc:407] Connection header received
0929 03:21:56.499622 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:21:56.499623 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:21:56.499633 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:21:56.499698 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:21:56.499984 (+ 286us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57702: BlockingRecv error: recv got EOF from 127.0.0.1:57702 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260929 03:21:56.630016 11452 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:21:56.629293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57710 (local address 127.20.178.125:42593)
0929 03:21:56.629382 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0929 03:21:56.629384 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:21:56.629495 (+ 111us) server_negotiation.cc:407] Connection header received
0929 03:21:56.629561 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:21:56.629562 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:21:56.629572 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:21:56.629622 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:21:56.629950 (+ 328us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57710: BlockingRecv error: recv got EOF from 127.0.0.1:57710 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260929 03:21:56.887838 11452 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:21:56.887242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57716 (local address 127.20.178.125:42593)
0929 03:21:56.887325 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0929 03:21:56.887327 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:21:56.887336 (+ 9us) server_negotiation.cc:407] Connection header received
0929 03:21:56.887371 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:21:56.887372 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:21:56.887382 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:21:56.887443 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:21:56.887785 (+ 342us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57716: BlockingRecv error: recv got EOF from 127.0.0.1:57716 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 03:21:57.253952 11540 catalog_manager.cc:1259] Loaded cluster ID: b27af727351241bfa0570d81f9c38165
I20260929 03:21:57.254031 11540 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 94d47136ff5e4b118b6c7bc5fba84bae: loading cluster ID for follower catalog manager: success
I20260929 03:21:57.254931 11540 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 94d47136ff5e4b118b6c7bc5fba84bae: acquiring CA information for follower catalog manager: success
I20260929 03:21:57.255231 11540 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 94d47136ff5e4b118b6c7bc5fba84bae: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 03:21:57.322894 11587 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b2f4204b0ccd4581a2b7838e1458d704: acquiring CA information for follower catalog manager: success
I20260929 03:21:57.323277 11587 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b2f4204b0ccd4581a2b7838e1458d704: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 03:21:57.403554 11655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:57.401195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57726 (local address 127.20.178.125:42593)
0929 03:21:57.401310 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0929 03:21:57.401311 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 03:21:57.401321 (+ 10us) server_negotiation.cc:407] Connection header received
0929 03:21:57.401375 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:21:57.401376 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:21:57.401430 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:21:57.401515 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:21:57.401647 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:57.402149 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:57.402857 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:57.403049 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:57.403099 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:21:57.403257 (+ 158us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:21:57.403366 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:21:57.403438 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0929 03:21:57.403457 (+ 19us) server_negotiation.cc:299] Negotiation successful
0929 03:21:57.403476 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88,"thread_start_us":67,"threads_started":1}
I20260929 03:21:57.407773 11661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:57.405426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45654 (local address 127.20.178.124:37121)
0929 03:21:57.405546 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0929 03:21:57.405548 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:21:57.405575 (+ 27us) server_negotiation.cc:407] Connection header received
0929 03:21:57.405618 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:21:57.405619 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:21:57.405653 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:21:57.405732 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:21:57.405825 (+ 93us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:57.406458 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:57.407127 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:57.407304 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:57.407351 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:21:57.407462 (+ 111us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:21:57.407553 (+ 91us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:21:57.407606 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0929 03:21:57.407700 (+ 94us) server_negotiation.cc:299] Negotiation successful
0929 03:21:57.407714 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":81,"threads_started":1}
I20260929 03:21:57.412302 11655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:57.409394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57736 (local address 127.20.178.125:42593)
0929 03:21:57.409438 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0929 03:21:57.409441 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:21:57.409623 (+ 182us) server_negotiation.cc:407] Connection header received
0929 03:21:57.409706 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:21:57.409707 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:21:57.409721 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:21:57.409777 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:21:57.409908 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:57.410640 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:57.411610 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:57.411832 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:57.411908 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:21:57.412057 (+ 149us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:21:57.412087 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:21:57.412132 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0929 03:21:57.412186 (+ 54us) server_negotiation.cc:299] Negotiation successful
0929 03:21:57.412205 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260929 03:21:57.412432 11661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:57.409458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45670 (local address 127.20.178.124:37121)
0929 03:21:57.409494 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0929 03:21:57.409496 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:21:57.409782 (+ 286us) server_negotiation.cc:407] Connection header received
0929 03:21:57.409858 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:21:57.409859 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:21:57.409869 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:21:57.409915 (+ 46us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:21:57.410089 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:57.410773 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:57.411600 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:57.411886 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:57.411923 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:21:57.412133 (+ 210us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:21:57.412158 (+ 25us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:21:57.412186 (+ 28us) server_negotiation.cc:1035] Waiting for connection context
0929 03:21:57.412339 (+ 153us) server_negotiation.cc:299] Negotiation successful
0929 03:21:57.412359 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260929 03:21:57.412541 11666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:21:57.409394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48686 (local address 127.20.178.126:39889)
0929 03:21:57.409555 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0929 03:21:57.409557 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:21:57.409641 (+ 84us) server_negotiation.cc:407] Connection header received
0929 03:21:57.409697 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:21:57.409698 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:21:57.409748 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:21:57.409836 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:21:57.409974 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:57.410449 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:57.411513 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:21:57.411818 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:21:57.411886 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:21:57.412067 (+ 181us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:21:57.412167 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:21:57.412228 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0929 03:21:57.412412 (+ 184us) server_negotiation.cc:299] Negotiation successful
0929 03:21:57.412438 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":50,"threads_started":1}
I20260929 03:21:57.414547 21193 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuoaHbX/build/release/bin/kudu with pid 11305
I20260929 03:21:57.420085 21193 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuoaHbX/build/release/bin/kudu with pid 11380
I20260929 03:21:57.424817 21193 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuoaHbX/build/release/bin/kudu with pid 11455
2026-09-29T03:21:57Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2778 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5533 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (264955 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST